diff --git a/README.md b/README.md index 2651d2f4a..71e8f1d83 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,16 @@ ## Modules - [hugegraph-loader](./hugegraph-loader): Loading datasets into the HugeGraph from multiple data sources. -- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and analysis dashboard (Include: data loading, schema management, graph traverser and display). We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow [this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start `hubble` with `server`. +- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and analysis dashboard (Include: data loading, schema management, graph traverser and display). - [hugegraph-tools](./hugegraph-tools): Command line tool for deploying, managing and backing-up/restoring graphs from HugeGraph. - [hugegraph-client](./hugegraph-client): A Java-written client for HugeGraph, providing `RESTful` APIs for accessing graph vertex/edge/schema/gremlin/variables and traversals etc. +## Usage + +- [hugegraph-loader](./hugegraph-loader): We can use `docker run -itd --name loader hugegraph/loader` to quickly start [loader](https://hub.docker.com/r/hugegraph/loader) or we can follow [this](./hugegraph-loader/README.md#212-docker-compose) to use docker-compose to start `loader` with `server`. And we can find more details in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/). +- [hugegraph-hubble](./hugegraph-hubble): We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow [this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start `hubble` with `server`. And we can find more details in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-hubble/). +- [hugegraph-client](./hugegraph-client): We can follow the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-client/) to learn how to quick start with `client`. + ## Maven Dependencies You could use import the dependencies in `maven` like this: diff --git a/hugegraph-hubble/README.md b/hugegraph-hubble/README.md index e19fd2804..6de7f6860 100644 --- a/hugegraph-hubble/README.md +++ b/hugegraph-hubble/README.md @@ -20,8 +20,7 @@ We can quickly start `hubble` in two ways: 1. We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble). 2. Or we can use the `docker-compose.yml` to start `hubble` with `hugegraph-server`. If we set `PRELOAD=true`, we can preload the example graph when starting `hugegraph-server`: - - ``` + ```yaml version: '3' services: server: diff --git a/hugegraph-loader/Dockerfile b/hugegraph-loader/Dockerfile new file mode 100644 index 000000000..06985e700 --- /dev/null +++ b/hugegraph-loader/Dockerfile @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +FROM maven:3.9.0-eclipse-temurin-11 AS build + +COPY . /pkg +WORKDIR /pkg + +RUN set -x \ + && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp + + +RUN set -x \ + && cd /pkg/hugegraph-loader/ \ + && echo "$(ls)" \ + && mvn clean package -DskipTests + + +FROM openjdk:11-slim + +COPY --from=build /pkg/hugegraph-loader/apache-hugegraph-loader-incubating-*/ /loader +WORKDIR /loader/ + +RUN set -x \ + && apt-get -q update \ + && apt-get -q install -y --no-install-recommends --no-install-suggests \ + dumb-init \ + procps \ + curl \ + lsof \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +VOLUME /loader + +ENTRYPOINT ["/usr/bin/dumb-init", "--"] +CMD ["tail","-f","/dev/null"] diff --git a/hugegraph-loader/README.md b/hugegraph-loader/README.md index 036b56bf3..e81c2b2f1 100644 --- a/hugegraph-loader/README.md +++ b/hugegraph-loader/README.md @@ -7,7 +7,7 @@ hugegraph-loader is a customizable command line utility for loading small to medium size graph datasets into the HugeGraph database from multiple data sources with various input formats. -## Features +## 1. Features - Multiple data sources, such as local file(path), HDFS file(path), MySQL - Various input formats, such as json, csv, and text with any delimiters. @@ -15,7 +15,123 @@ hugegraph-loader is a customizable command line utility for loading small to med - Detecting schema from data automatically, reduce the complex work of schema management. - Advanced customized operations with groovy script, users can configure how to construct vertices and edges by themselves. -## Building +## 2. Usage for Docker(Recommand) + +- Run `loader` with Docker + - Docker run + - Docker-compose +- Load data in docker container `loader` + +### 2.1 Start with Docker + +#### 2.1.1 Docker run + +Use the command `docker run -itd --name loader hugegraph/loader` to start loader. + +If you want to load your data, you can mount the data folder like `-v /path/to/data/file:/loader/file` + + +#### 2.1.2 Docker-compose + +The example `docker-compose.yml` is [here](./docker/example/docker-compose.yml) + +If you want to load your data, you can mount the data folder like: +```yaml +volumes: + - /path/to/data/file:/loader/file +``` + +Use the command `docker-compose up -d` to deploy `loader` with `server` and `hubble`. + +### 2.2 Load data with docker container + +#### 2.2.1 load data with docker + +> If the `loader` and `server` is in the same docker network (for example, you deploy `loader` and `server` with `docker-compose`), we can set `-h {server_container_name}`. In our example, the container name of `server` is `graph` +> +> If `loader` is deployed alone, the `-h` should be set to the ip of the host of `server`. Other parameter description is [here](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/#341-parameter-description) + +```bash +docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080 +``` + +Then we can see the result. + +```bash +HugeGraphLoader worked in NORMAL MODE +vertices/edges loaded this time : 8/6 +-------------------------------------------------- +count metrics + input read success : 14 + input read failure : 0 + vertex parse success : 8 + vertex parse failure : 0 + vertex insert success : 8 + vertex insert failure : 0 + edge parse success : 6 + edge parse failure : 0 + edge insert success : 6 + edge insert failure : 0 +-------------------------------------------------- +meter metrics + total time : 0.199s + read time : 0.046s + load time : 0.153s + vertex load time : 0.077s + vertex load rate(vertices/s) : 103 + edge load time : 0.112s + edge load rate(edges/s) : 53 +``` + +Then you can use `curl` or `hubble` to see the result. + +```bash +> curl "http://localhost:8080/graphs/hugegraph/graph/vertices" | gunzip +{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"1:li,nary","label":"person","type":"vertex","properties":{"name":"li,nary","age":26,"city":"Wu,han"}}]} +``` + +If you want to check the edges, use `curl "http://localhost:8080/graphs/hugegraph/graph/edges" | gunzip` + +#### 2.2.2 enter the docker container to load data + +If you want to do some additional operation in the container, you can enter the container as follows: + +```bash +docker exec -it loader bash +``` + +Then, you can load data as follows: + +```bash +sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080 +``` + +The result is as same as above. + +## 3. Use loader directly + +> notice: currently, version is `1.0.0` + +Download and unzip the compiled archive + +```bash +wget https://downloads.apache.org/incubator/hugegraph/{version}/apache-hugegraph-toolchain-incubating-{version}.tar.gz +tar zxf *hugegraph*.tar.gz +``` + +Then, load data with example file: + +```bash +cd apache-hugegraph-toolchain-incubating-{version} +cd apache-hugegraph-loader-incubating-{version} +sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy +``` + +More details is in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/) + +## 4. Building + +You can also build the `loader` by yourself. Required: @@ -34,10 +150,10 @@ To build with default tests: mvn clean install ``` -## Doc +## 5. Doc The [loader homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/) contains more information about it. -## License +## 6. License hugegraph-loader is licensed under Apache 2.0 License. diff --git a/hugegraph-loader/docker/example/docker-compose.yml b/hugegraph-loader/docker/example/docker-compose.yml new file mode 100644 index 000000000..7c9a1b906 --- /dev/null +++ b/hugegraph-loader/docker/example/docker-compose.yml @@ -0,0 +1,37 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: '3' +services: + server: + image: hugegraph/hugegraph + container_name: graph + ports: + - 8080:8080 + + hubble: + image: hugegraph/hubble + container_name: hubble + ports: + - 8088:8088 + + loader: + image: hugegraph/loader + container_name: loader + # mount your own data here + # volumes: + # - /path/to/data/file:/loader/file