diff --git a/.gitattributes b/.gitattributes index ffb061de0..ef04141f9 100755 --- a/.gitattributes +++ b/.gitattributes @@ -10,5 +10,6 @@ apache-release.sh export-ignore # ignored directory .github/ export-ignore hugegraph-dist/scripts/ export-ignore +.serena/ export-ignore # only exclude the root /assembly/ export-ignore diff --git a/.licenserc.yaml b/.licenserc.yaml index da6493d7f..397cb7e53 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -56,6 +56,7 @@ header: # `header` section is configurations for source codes license header. - 'LICENSE' - 'NOTICE' - 'DISCLAIMER' + - '.serena/**' - '**/*.md' - '**/*.versionsBackup' - '**/*.log' diff --git a/.serena/.gitignore b/.serena/.gitignore new file mode 100644 index 000000000..14d86ad62 --- /dev/null +++ b/.serena/.gitignore @@ -0,0 +1 @@ +/cache diff --git a/.serena/memories/README_INDEX.md b/.serena/memories/README_INDEX.md new file mode 100644 index 000000000..3f506906f --- /dev/null +++ b/.serena/memories/README_INDEX.md @@ -0,0 +1,244 @@ +# Memory Index - HugeGraph Toolchain Project + +## Onboarding Complete ✓ + +This project has been successfully initialized with Serena MCP. Below is an index of all available memory files. + +## Available Memories (7 Core Files) + +### Core Project Information + +1. **project_overview.md** (125 lines) + - Project purpose and goals + - All 6 modules (client, loader, hubble, tools, client-go, spark-connector) + - Technology stack (Java 8, Node.js 18.20.8, Go, React, Spring Boot) + - Module dependencies and relationships + - External dependencies + - Project structure + +### Code Quality and Style + +2. **code_style_and_conventions.md** (311 lines) + - Java code style (indentation, naming, formatting) + - Naming conventions (classes, methods, variables) + - Import rules and prohibited imports + - Maven POM style + - Frontend code style (TypeScript, CSS/Less) + - Go code style + - Design patterns used in each module + - File organization standards + - Commit message format + +3. **task_completion_checklist.md** (372 lines) ⭐ **IMPORTANT** + - Code quality checks before committing + - Testing requirements + - Build verification steps + - Documentation update checklist + - Git pre-commit checklist + - Pull request checklist + - CI/CD pipeline details + - Common issues and solutions + - Release-specific tasks + +### Architecture and Design + +4. **architecture_and_design_patterns.md** (571 lines) + - Overall system architecture + - Module-specific architectures: + - hugegraph-client: Layered architecture, Manager pattern + - hugegraph-loader: Pipeline architecture + - hugegraph-hubble: Frontend (React+MobX) + Backend (Spring Boot) + - hugegraph-tools: Command pattern + - Design patterns (Factory, Builder, Strategy, Observer, Repository) + - Cross-cutting concerns (error handling, logging) + - Configuration management + +### Testing + +5. **testing_infrastructure.md** (634 lines) + - Testing philosophy (unit, integration, functional) + - Test organization and structure + - Module-specific testing: + - hugegraph-client: UnitTestSuite, ApiTestSuite, FuncTestSuite + - hugegraph-loader: Test profiles (unit, file, hdfs, jdbc, kafka) + - hugegraph-hubble: Backend (Spring Test) + Frontend (Jest) + - hugegraph-client-go: Go standard testing + - CI/CD testing pipelines + - Test coverage tools and targets + - Common testing patterns + - Debugging tests + +### Development Workflows + +6. **common_development_workflows.md** (657 lines) + - Daily development workflows: + - Starting new features + - Fixing bugs + - Adding tests + - Refactoring code + - Module-specific workflows + - Troubleshooting common issues + - Release workflow + - Useful development commands + - Git hooks setup + - IDE configuration (IntelliJ IDEA, VS Code) + +## Quick Start Guide + +### For New Developers + +1. **Read First**: + - `project_overview.md` - Understand what the project is + - `common_development_workflows.md` - Learn essential commands and workflows + +2. **Before Making Changes**: + - `code_style_and_conventions.md` - Learn coding standards + - `task_completion_checklist.md` - Know what to check before committing + +3. **When Working on Code**: + - `architecture_and_design_patterns.md` - Understand design patterns + +4. **When Writing Tests**: + - `testing_infrastructure.md` - Learn testing approach + +### For System Setup + +**Prerequisites** (macOS): +```bash +# Java 11 (required) +/usr/libexec/java_home -V +export JAVA_HOME=$(/usr/libexec/java_home -v 11) + +# Maven +brew install maven + +# Node.js 18.20.8 (for Hubble) +nvm install 18.20.8 +nvm use 18.20.8 +npm install -g yarn + +# Python 3 (for Hubble build) +brew install python3 +pip3 install -r hugegraph-hubble/hubble-dist/assembly/travis/requirements.txt +``` + +**Build Entire Project**: +```bash +mvn clean install -DskipTests -Dmaven.javadoc.skip=true -ntp +``` + +**Run Tests**: +```bash +# Client tests +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# Loader tests +cd hugegraph-loader +mvn test -P unit -ntp + +# Hubble tests +cd hugegraph-hubble/hubble-fe +yarn test +``` + +## Essential Commands Cheat Sheet + +### Build Commands +```bash +# Full project +mvn clean install -DskipTests -Dmaven.javadoc.skip=true -ntp + +# Specific module (e.g., client) +mvn install -pl hugegraph-client -am -DskipTests -ntp + +# Hubble (requires dependencies built first) +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -ntp +``` + +### Testing Commands +```bash +# Client unit tests +cd hugegraph-client && mvn test -Dtest=UnitTestSuite -ntp + +# Loader tests +cd hugegraph-loader && mvn test -P unit -ntp + +# Single test +mvn test -Dtest=ClassName#methodName -ntp +``` + +### Code Quality +```bash +# Checkstyle +mvn checkstyle:check + +# License check +mvn apache-rat:check + +# EditorConfig validation +mvn editorconfig:check +``` + +### Git Commands (IMPORTANT: Always use --no-pager) +```bash +# View history +git --no-pager log --oneline -10 + +# View changes +git --no-pager diff HEAD~1 +``` + +**See `common_development_workflows.md` for complete command reference** + +## Key Project Facts + +- **Language**: Java 8 (main), Go, TypeScript +- **Build Tool**: Maven 3.x +- **Test Framework**: JUnit 4 + Mockito +- **Frontend**: React + TypeScript + MobX (Node.js 18.20.8) +- **Backend**: Spring Boot +- **Version**: 1.7.0 +- **License**: Apache 2.0 +- **Repository**: https://github.com/apache/hugegraph-toolchain + +## Common Pitfalls to Avoid + +1. ❌ **DON'T** use `git log` without `--no-pager` flag +2. ❌ **DON'T** commit without running checkstyle and tests +3. ❌ **DON'T** use star imports (`import org.apache.*`) +4. ❌ **DON'T** use `System.out.println` (use logger instead) +5. ❌ **DON'T** forget Apache 2.0 license headers +6. ❌ **DON'T** use tabs (use 4 spaces for Java, 2 for frontend) +7. ❌ **DON'T** exceed 100 character line length +8. ❌ **DON'T** commit code that fails CI checks + +## Getting Help + +- **Documentation**: https://hugegraph.apache.org/docs/ +- **Issues**: https://github.com/apache/hugegraph-toolchain/issues +- **Mailing List**: dev@hugegraph.apache.org +- **Memory Files**: Check `.serena/memories/` directory + +## Memory Statistics + +- **Total Memory Files**: 7 (including this index) +- **Total Lines**: ~2,900+ +- **Total Size**: ~85KB +- **Coverage Areas**: + - Project overview and structure + - Code style and conventions + - Architecture and design patterns + - Testing infrastructure + - Development workflows + - Task completion checklists + +## Last Updated + +Onboarding completed: 2025-11-05 + +--- + +**Note**: All memories are stored in `.serena/memories/` directory and can be read using Serena MCP tools. diff --git a/.serena/memories/architecture_and_design_patterns.md b/.serena/memories/architecture_and_design_patterns.md new file mode 100644 index 000000000..1861c5c5b --- /dev/null +++ b/.serena/memories/architecture_and_design_patterns.md @@ -0,0 +1,572 @@ +# Architecture and Design Patterns - HugeGraph Toolchain + +## Overall Architecture + +### System Context +``` +┌─────────────────────────────────────────────────────────────┐ +│ HugeGraph Ecosystem │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────┐ ┌──────────────────────────────┐ │ +│ │ HugeGraph │◄─────┤ HugeGraph Toolchain │ │ +│ │ Server │ │ │ │ +│ │ (Core) │ │ ┌─────────────────────────┐ │ │ +│ └──────────────┘ │ │ hugegraph-client │ │ │ +│ ▲ │ │ (RESTful API wrapper) │ │ │ +│ │ │ └──────────┬──────────────┘ │ │ +│ │ │ │ │ │ +│ │ │ ┌──────▼──────────┐ │ │ +│ REST API │ │ ┌──────────────┐ │ │ +│ (HTTP/HTTPS) │ │ │ loader │ │ │ +│ │ │ │ │ tools │ │ │ +│ │ │ │ │ hubble-be │ │ │ +│ │ │ │ │ spark │ │ │ +│ └──────────────┼──────┘ │ client-go │ │ │ +│ │ └──────────────┘ │ │ +│ │ │ │ +│ ┌──────────────┐ │ ┌──────────────────┐ │ │ +│ │ External │────►│ │ hubble-fe │ │ │ +│ │ Data Sources │ │ │ (React Web UI) │ │ │ +│ │ (CSV/HDFS/ │ │ └──────────────────┘ │ │ +│ │ JDBC/Kafka) │ │ │ │ +│ └──────────────┘ └──────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Module-Specific Architectures + +## 1. hugegraph-client Architecture + +### Layered Architecture +``` +┌─────────────────────────────────────────────┐ +│ Application Layer │ +│ (User code using HugeGraph client) │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ Manager Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │SchemaManager │ │GraphManager │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │TraverserMgr │ │JobManager │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │TaskManager │ │AuthManager │ │ +│ └──────────────┘ └──────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ API Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ VertexAPI │ │ EdgeAPI │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │ SchemaAPI │ │ GremlinAPI │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │ TraverserAPI │ │ JobAPI │ │ +│ └──────────────┘ └──────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ REST Client Layer │ +│ ┌──────────────────────────────┐ │ +│ │ RestClient │ │ +│ │ - HTTP connection pool │ │ +│ │ - Request/Response handling │ │ +│ │ - Authentication │ │ +│ │ - Error handling │ │ +│ └──────────────────────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ HugeGraph Server (REST API) │ +└─────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. RestClient (Core) +**Responsibilities**: +- HTTP/HTTPS connection management +- Request serialization (Java objects → JSON) +- Response deserialization (JSON → Java objects) +- Authentication (Basic Auth, Token) +- Error handling and retry logic +- Connection pooling + +**Key Methods**: +```java +// Generic request methods +public T get(String path, Class responseClass) +public T post(String path, Object request, Class responseClass) +public T put(String path, Object request, Class responseClass) +public T delete(String path, Class responseClass) + +// With custom headers +public T request(HttpMethod method, String path, Object request, + Map headers, Class responseClass) +``` + +#### 2. Manager Pattern +Each manager handles a specific domain: + +**SchemaManager**: Schema CRUD operations +```java +// Get manager +SchemaManager schema = hugegraph.schema(); + +// Operations +schema.propertyKey("name").asText().create(); +schema.vertexLabel("person").properties("name", "age").create(); +schema.edgeLabel("knows").link("person", "person").create(); +schema.indexLabel("personByName").onV("person").by("name").create(); +``` + +**GraphManager**: Vertex/Edge operations +```java +GraphManager graph = hugegraph.graph(); + +// CRUD +Vertex v = graph.addVertex("person", "name", "Alice", "age", 30); +Edge e = v.addEdge("knows", target, "date", "2023-01-01"); +Vertex retrieved = graph.getVertex(id); +graph.removeVertex(id); +``` + +**TraverserManager**: Graph algorithms +```java +TraverserManager traverser = hugegraph.traverser(); + +// Algorithms +Path shortestPath = traverser.shortestPath(sourceId, targetId, direction, maxDepth); +List kHop = traverser.kHop(sourceId, direction, depth); +List kShortestPaths = traverser.kShortestPaths(sourceId, targetId, k); +``` + +#### 3. Builder Pattern (Fluent API) +```java +// PropertyKey builder +PropertyKey age = schema.propertyKey("age") + .asInt() + .valueSingle() // Single value (not set) + .ifNotExist() // Create only if not exists + .create(); + +// VertexLabel builder +VertexLabel person = schema.vertexLabel("person") + .properties("name", "age", "city") + .primaryKeys("name") // Required fields + .nullableKeys("city") // Optional fields + .ifNotExist() + .create(); + +// EdgeLabel builder +EdgeLabel knows = schema.edgeLabel("knows") + .sourceLabel("person") + .targetLabel("person") + .properties("date", "weight") + .frequency(Frequency.SINGLE) // One edge per (source,target) pair + .ifNotExist() + .create(); +``` + +### Serialization Layer +**Purpose**: Convert between Java objects and JSON + +**Key Classes**: +- `VertexSerializer`: Serialize/deserialize vertices +- `EdgeSerializer`: Serialize/deserialize edges +- `PathSerializer`: Serialize/deserialize paths +- `ResultDeserializer`: Generic result parsing + +## 2. hugegraph-loader Architecture + +### Pipeline Architecture +``` +┌──────────────────────────────────────────────────────────┐ +│ Data Loading Pipeline │ +└──────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 1: Data Source Connection │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Source Factory (based on SourceType) │ │ +│ │ - FileSource (CSV, JSON, TXT) │ │ +│ │ - HDFSSource (HDFS files) │ │ +│ │ - JDBCSource (MySQL, PostgreSQL, Oracle) │ │ +│ │ - KafkaSource (Kafka topics) │ │ +│ └─────────────────────────────────────────────────┘ │ +└──────────────────┬───────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 2: Data Reading & Parsing │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Fetcher (source-specific) │ │ +│ │ - FileFetcher: Read file line-by-line │ │ +│ │ - JDBCFetcher: Execute SQL query │ │ +│ │ - KafkaFetcher: Consume messages │ │ +│ └────────────────┬────────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Reader (format-specific) │ │ +│ │ - CSVReader: Parse CSV records │ │ +│ │ - JSONReader: Parse JSON objects │ │ +│ │ - TextReader: Parse text lines │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 3: Element Building │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Mapping Config (struct.json) │ │ +│ │ - Field mappings: source → graph property │ │ +│ │ - ID generation strategies │ │ +│ │ - Value conversions │ │ +│ └────────────────┬────────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ ElementBuilder │ │ +│ │ - Build Vertex from row/record │ │ +│ │ - Build Edge from row/record │ │ +│ │ - Apply transformations │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 4: Batch Insertion │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ InsertTask (multi-threaded) │ │ +│ │ - Buffer elements (batch size: 500 default) │ │ +│ │ - Bulk insert via hugegraph-client API │ │ +│ │ - Error handling & retry logic │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ HugeGraph Server │ +└──────────────────────────────────────────────────────────┘ +``` + +### Key Design Patterns + +#### 1. Factory Pattern (Source Creation) +```java +public interface Source { + Fetcher createFetcher(); +} + +// Factory method +public static Source create(SourceType type, SourceConfig config) { + switch (type) { + case FILE: + return new FileSource(config); + case HDFS: + return new HDFSSource(config); + case JDBC: + return new JDBCSource(config); + case KAFKA: + return new KafkaSource(config); + default: + throw new IllegalArgumentException(); + } +} +``` + +#### 2. Strategy Pattern (ID Generation) +Different strategies for generating vertex/edge IDs: +- `PrimaryKeyIdStrategy`: Use primary key fields +- `CustomIdStrategy`: User-defined ID field +- `AutomaticIdStrategy`: Server-generated IDs + +#### 3. Template Method Pattern (Parsing) +```java +abstract class AbstractReader { + // Template method + public final List read() { + open(); + List records = parseRecords(); + close(); + return records; + } + + protected abstract void open(); + protected abstract List parseRecords(); + protected abstract void close(); +} +``` + +## 3. hugegraph-hubble Architecture + +### Frontend Architecture (React + MobX) +``` +┌──────────────────────────────────────────────────────────┐ +│ Hubble Frontend │ +├──────────────────────────────────────────────────────────┤ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Presentation Layer (React Components) │ │ +│ │ │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphManager │ │ DataAnalyze │ │ │ +│ │ │ Pages │ │ Pages │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ SchemaManage │ │ DataImport │ │ │ +│ │ │ Pages │ │ Pages │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ State Management Layer (MobX Stores) │ │ +│ │ │ │ +│ │ ┌──────────────────────┐ │ │ +│ │ │ GraphManagementStore │ (graph connections) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ DataAnalyzeStore │ (query & analysis) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ SchemaStore │ (schema operations) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ DataImportStore │ (data loading) │ │ +│ │ └──────────────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ API Service Layer │ │ +│ │ (HTTP requests to backend) │ │ +│ └────────────────┬───────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ HTTP/REST API +┌──────────────────────────────────────────────────────────┐ +│ Hubble Backend (Spring Boot) │ +├──────────────────────────────────────────────────────────┤ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Controller Layer │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphConn │ │ Schema │ │ │ +│ │ │ Controller │ │ Controller │ │ │ +│ │ ├──────────────┤ ├──────────────┤ │ │ +│ │ │ Query │ │ DataImport │ │ │ +│ │ │ Controller │ │ Controller │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Service Layer (Business Logic) │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphConn │ │ Schema │ │ │ +│ │ │ Service │ │ Service │ │ │ +│ │ ├──────────────┤ ├──────────────┤ │ │ +│ │ │ Query │ │ DataImport │ │ │ +│ │ │ Service │ │ Service │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Repository Layer (Data Persistence) │ │ +│ │ - File-based storage (local disk) │ │ +│ │ - Graph connection metadata │ │ +│ └────────────────┬───────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ REST API (via hugegraph-client) +┌──────────────────────────────────────────────────────────┐ +│ HugeGraph Server │ +└──────────────────────────────────────────────────────────┘ +``` + +### Key Design Patterns (Hubble) + +#### 1. Observer Pattern (MobX) +```typescript +// Store definition +class GraphManagementStore { + @observable currentGraph: GraphConnection | null = null; + @observable graphList: GraphConnection[] = []; + + @action + async loadGraphs() { + const response = await api.getGraphs(); + this.graphList = response.data; + } + + @computed + get activeGraphName() { + return this.currentGraph?.name || 'None'; + } +} + +// Component observing store +@observer +class GraphSelector extends React.Component { + render() { + const { graphStore } = this.props; + return
{graphStore.activeGraphName}
; + } +} +``` + +#### 2. Repository Pattern (Backend) +```java +// Entity +@Entity +public class GraphConnection { + @Id + private Long id; + private String name; + private String host; + private Integer port; + // ... +} + +// Repository interface +public interface GraphConnectionRepository { + GraphConnection save(GraphConnection connection); + GraphConnection findById(Long id); + List findAll(); + void deleteById(Long id); +} + +// Service using repository +@Service +public class GraphConnectionService { + @Autowired + private GraphConnectionRepository repository; + + public GraphConnection create(GraphConnection connection) { + return repository.save(connection); + } +} +``` + +## 4. hugegraph-tools Architecture + +### Command Pattern +``` +┌─────────────────────────────────────────┐ +│ CLI Entry Point │ +└───────────────┬─────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────┐ +│ Command Router │ +│ (parse args, dispatch command) │ +└───────────────┬─────────────────────────┘ + │ + ├─► backup (GraphBackupCommand) + ├─► restore (GraphRestoreCommand) + ├─► deploy (DeployCommand) + ├─► graph-list (GraphListCommand) + ├─► graph-clear (GraphClearCommand) + └─► graph-mode-set (GraphModeCommand) +``` + +**Command Interface**: +```java +public interface Command { + String name(); + void execute(String[] args); +} + +// Example implementation +public class BackupCommand implements Command { + public String name() { return "backup"; } + + public void execute(String[] args) { + // Parse options + String graph = parseGraphOption(args); + String directory = parseDirectoryOption(args); + + // Execute backup via client API + HugeClient client = createClient(); + client.graphs().backup(graph, directory); + } +} +``` + +## Cross-Cutting Concerns + +### Error Handling Strategy + +**Client/Loader/Tools**: +```java +try { + // Operation +} catch (ServerException e) { + // Server-side error (4xx, 5xx) + log.error("Server error: {}", e.getMessage()); + throw new LoaderException("Failed to load data", e); +} catch (ClientException e) { + // Client-side error (network, serialization) + log.error("Client error: {}", e.getMessage()); + throw new LoaderException("Client communication failed", e); +} +``` + +### Logging Strategy + +**All modules use Log4j2**: +```java +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class MyClass { + private static final Logger LOG = LogManager.getLogger(MyClass.class); + + public void method() { + LOG.debug("Debug message"); + LOG.info("Info message"); + LOG.warn("Warning message"); + LOG.error("Error message", exception); + } +} +``` + +### Configuration Management + +**Loader** uses JSON structure files: +```json +{ + "version": "2.0", + "vertices": [ + { + "label": "person", + "input": { + "type": "file", + "path": "data/persons.csv", + "format": "CSV" + }, + "mapping": { + "id": "id", + "properties": { + "name": "name", + "age": "age" + } + } + } + ] +} +``` + +**Hubble** uses Spring properties: +```properties +server.port=8088 +spring.application.name=hugegraph-hubble +graph.server.host=localhost +graph.server.port=8080 +``` diff --git a/.serena/memories/code_style_and_conventions.md b/.serena/memories/code_style_and_conventions.md new file mode 100644 index 000000000..0c14759c6 --- /dev/null +++ b/.serena/memories/code_style_and_conventions.md @@ -0,0 +1,312 @@ +# Code Style and Conventions for HugeGraph Toolchain + +## General Principles +- **Language**: English for all code, comments, and documentation +- **License**: All source files require Apache 2.0 license headers +- **Encoding**: UTF-8 for all files +- **Line Endings**: LF (Unix-style) +- **Final Newline**: Always insert final newline + +## Java Code Style + +### Basic Formatting +- **Indentation**: 4 spaces (NO TABS) +- **Continuation Indent**: 8 spaces +- **Line Length**: Maximum 100 characters +- **Line Wrapping**: Enabled for long lines +- **Blank Lines**: + - Keep max 1 blank line in declarations + - Keep max 1 blank line in code + - 1 blank line around classes + - 1 blank line after class header + +### Naming Conventions +- **Package Names**: `^[a-z]+(\.[a-z][a-z0-9]*)*$` + - Example: `org.apache.hugegraph.client` +- **Class Names**: `PascalCase` (e.g., `RestClient`, `GraphManager`) +- **Type Parameters**: `^[A-Z][a-zA-Z0-9]*$` (e.g., `T`, `K`, `V`) +- **Constants**: `UPPER_SNAKE_CASE` (e.g., `DEFAULT_TIMEOUT`, `MAX_RETRIES`) +- **Variables**: `camelCase` starting with lowercase (e.g., `vertexId`, `edgeLabel`) +- **Methods**: `camelCase` starting with lowercase, must have 2+ chars + - Pattern: `^[a-z][a-z0-9][a-zA-Z0-9_]*$` + - Example: `getVertexById()`, `createEdge()` +- **Parameters**: `camelCase` (e.g., `userId`, `timeout`) + +### Import Rules +- NO star imports (`import org.apache.*` forbidden) +- Remove unused imports +- Remove redundant imports +- Import order (configured in .editorconfig): + 1. Static imports + 2. `java.**` + 3. `javax.**` + 4. `org.**` + 5. `com.**` + 6. All others + +### Prohibited Imports (Checkstyle) +- `java.util.logging.Logging` +- `sun.misc.BASE64Encoder/Decoder` +- Shaded/internal packages from Hadoop, HBase, Netty, etc. +- `org.codehaus.jackson` (use `com.fasterxml.jackson` instead) +- `org.jetbrains.annotations` + +### Code Structure +- **Braces**: + - Always use braces for if/while/for (multi-line) + - `do-while` always requires braces + - Opening brace on same line (K&R style) +- **Whitespace**: + - No whitespace before: `,`, `;`, `.`, post-increment/decrement + - Whitespace around operators: `=`, `+`, `-`, `*`, `/`, etc. + - Proper padding in parentheses +- **Empty Blocks**: Only `{}` allowed (not `{ }`) + +### Java-Specific Rules +- **Array Style**: `String[] args` (NOT `String args[]`) +- **Generic Whitespace**: Follow standard Java conventions +- **Equals/HashCode**: Must implement both or neither +- **Switch Statement**: Must have `default` case +- **Finalize**: No finalizers allowed +- **System.out.println**: PROHIBITED in source code (use logger) + +### Comments and JavaDoc +- **Line Comments**: Not at first column, use proper indentation +- **JavaDoc**: + - Add `

` tag on empty lines + - Do not wrap if one line + - Comment indentation: 4 spaces + +### Annotations +- Each annotation on separate line (for methods/constructors) +- Single parameterless annotation allowed on same line (other contexts) + +## Maven POM Style + +### XML Formatting +- **Indentation**: 4 spaces +- **Line Length**: Maximum 120 characters +- **Text Wrap**: Off for XML +- **Empty Tags**: Space inside (``) + +### POM Organization +```xml + + + + + + + + + + + + + + + + + + + + + + + +``` + +## Frontend Code Style (Hubble) + +### TypeScript/JavaScript +- **Formatter**: Prettier (configured in `.prettierrc`) +- **Linter**: ESLint/TSLint +- **Naming**: + - Components: PascalCase (`GraphViewer.tsx`) + - Files: kebab-case or PascalCase + - Variables: camelCase + +### CSS/Less +- **Linter**: Stylelint (configured in `.stylelintrc`) +- **Naming**: kebab-case for class names +- **Indentation**: 2 spaces + +### Pre-commit Hooks +- **Husky**: Runs on git commit +- **lint-staged**: Auto-format staged files +- Configuration: `.lintstagedrc.yml` + +## Go Code Style (client-go) + +### Standard Go Conventions +- Follow official Go formatting (`gofmt`) +- Use `go vet` for static analysis +- Run tests with race detector: `go test -race` + +### Naming +- Exported names: Start with uppercase +- Unexported names: Start with lowercase +- Package names: Short, lowercase, single word + +## Design Patterns and Architecture + +### hugegraph-client Patterns + +#### Manager Pattern +Separate managers for different API domains: +```java +// Schema operations +SchemaManager schemaManager = hugegraph.schema(); + +// Graph operations +GraphManager graphManager = hugegraph.graph(); + +// Traversal algorithms +TraverserManager traverser = hugegraph.traverser(); + +// Async jobs +JobManager jobManager = hugegraph.job(); + +// Authentication +AuthManager authManager = hugegraph.auth(); +``` + +#### Builder Pattern +Fluent API for constructing schema elements: +```java +VertexLabel person = schema.vertexLabel("person") + .properties("name", "age", "city") + .primaryKeys("name") + .nullableKeys("city") + .create(); +``` + +#### RESTful Wrapper +- `RestClient`: Base HTTP communication layer +- All API classes extend or use `RestClient` +- Consistent error handling with custom exceptions + +### hugegraph-loader Patterns + +#### Pipeline Architecture +``` +Source → Parser → Transformer → Builder → BatchInserter → HugeGraph +``` + +- **ParseTask**: Read and parse data from sources +- **InsertTask**: Batch insert into HugeGraph +- **ElementBuilder**: Construct vertices/edges from raw data + +#### Source Abstraction +Unified interface for different data sources: +```java +interface Source { + Fetcher createFetcher(); +} + +// Implementations: +- FileSource (CSV, JSON, TXT) +- HDFSSource +- JDBCSource +- KafkaSource +``` + +### hugegraph-hubble Patterns + +#### Frontend Architecture +- **Store Pattern**: MobX stores for state management + - `GraphManagementStore`: Graph connection management + - `DataAnalyzeStore`: Query and analysis state + - `SchemaStore`: Schema management state +- **Component Hierarchy**: Container → Component → Sub-component + +#### Backend Architecture (Spring Boot) +- **Controller**: HTTP request handling +- **Service**: Business logic layer +- **Repository**: Data persistence (local file-based) +- **DTO/Entity**: Data transfer and domain objects + +## File Organization + +### Java Package Structure +``` +org.apache.hugegraph/ +├── api/ # RESTful API implementations +├── client/ # Client interfaces and implementations +├── driver/ # Driver layer +├── structure/ # Graph structure elements (Vertex, Edge, etc.) +├── exception/ # Custom exceptions +├── serializer/ # JSON serialization/deserialization +├── util/ # Utility classes +└── version/ # Version information +``` + +### Test Organization +``` +src/test/java/ +├── unit/ # Unit tests (no external dependencies) +├── api/ # API integration tests (require server) +└── functional/ # End-to-end functional tests +``` + +## Version Control Practices + +### Commit Messages +- Format: `type(scope): subject` +- Types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore` +- Examples: + - `feat(client): add batch vertex query API` + - `fix(loader): handle empty CSV files correctly` + - `chore(hubble): update Node.js version to 18.20.8` + +### Branch Naming +- `master`: Main development branch +- `release-*`: Release branches +- `feature/*`: Feature branches +- `fix/*`: Bug fix branches + +## Testing Conventions + +### Test Class Naming +- Unit tests: `*Test.java` (e.g., `RestClientTest.java`) +- Test suites: `*TestSuite.java` (e.g., `UnitTestSuite.java`) + +### Test Method Naming +- Descriptive names: `testGetVertexById()`, `testCreateEdgeWithInvalidLabel()` +- Use `@Test` annotation (JUnit 4) + +### Test Organization +- Group tests into suites: + - `UnitTestSuite`: No external dependencies + - `ApiTestSuite`: API integration tests + - `FuncTestSuite`: Functional/E2E tests + +## Documentation Standards + +### JavaDoc Requirements +- All public APIs must have JavaDoc +- Include `@param`, `@return`, `@throws` tags +- Example usage in class-level JavaDoc + +### README Structure +```markdown +# Module Name + +## Features +## Quick Start +## Usage +## Doc +## License +``` + +## Error Handling + +### Java Exceptions +- Use custom exceptions: `HugeException`, `ServerException`, `ClientException` +- Proper exception chaining with causes +- Meaningful error messages + +### Go Error Handling +- Return errors explicitly: `func() (result, error)` +- Handle errors at call site +- Wrap errors with context: `fmt.Errorf("context: %w", err)` diff --git a/.serena/memories/common_development_workflows.md b/.serena/memories/common_development_workflows.md new file mode 100644 index 000000000..8fa64e610 --- /dev/null +++ b/.serena/memories/common_development_workflows.md @@ -0,0 +1,658 @@ +# Common Development Workflows - HugeGraph Toolchain + +## Daily Development Workflows + +### 1. Starting a New Feature + +**Step 1: Create Feature Branch** +```bash +# Update master +git checkout master +git pull origin master + +# Create feature branch +git checkout -b feature/add-batch-query-api +``` + +**Step 2: Make Changes** +```bash +# Edit code in your IDE +# Follow code style guidelines (see code_style_and_conventions.md) +``` + +**Step 3: Local Testing** +```bash +# Run unit tests +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# Run checkstyle +mvn checkstyle:check +``` + +**Step 4: Commit Changes** +```bash +git add . +git commit -m "feat(client): add batch query API for vertices" +``` + +**Step 5: Push and Create PR** +```bash +git push origin feature/add-batch-query-api +# Open PR on GitHub +``` + +### 2. Fixing a Bug + +**Step 1: Reproduce the Bug** +```bash +# Write a failing test first (TDD approach) +cd hugegraph-loader +vim src/test/java/org/apache/hugegraph/loader/test/functional/CSVLoadTest.java + +# Add test case +@Test +public void testHandleEmptyCSVFile() { + // Test that reproduces the bug +} + +# Run test - should fail +mvn test -Dtest=CSVLoadTest#testHandleEmptyCSVFile -ntp +``` + +**Step 2: Fix the Bug** +```bash +# Edit source code to fix the issue +vim src/main/java/org/apache/hugegraph/loader/reader/CSVReader.java +``` + +**Step 3: Verify Fix** +```bash +# Run test again - should pass +mvn test -Dtest=CSVLoadTest#testHandleEmptyCSVFile -ntp + +# Run all related tests +mvn test -P file +``` + +**Step 4: Commit with Issue Reference** +```bash +git add . +git commit -m "fix(loader): handle empty CSV files correctly + +Fixes #123 + +Previously, the loader would throw NullPointerException when +encountering empty CSV files. Now it gracefully skips empty files +and logs a warning." +``` + +### 3. Adding Tests for Existing Code + +**Step 1: Identify Coverage Gaps** +```bash +# Generate coverage report +mvn test jacoco:report + +# Open report +open target/site/jacoco/index.html + +# Find classes with low coverage +``` + +**Step 2: Write Tests** +```bash +# Create test class if doesn't exist +vim src/test/java/org/apache/hugegraph/client/RestClientTest.java +``` + +```java +public class RestClientTest { + @Test + public void testConnectionTimeout() { + // Test timeout handling + } + + @Test + public void testRetryOnNetworkError() { + // Test retry logic + } +} +``` + +**Step 3: Add to Test Suite** +```java +@RunWith(Suite.class) +@Suite.SuiteClasses({ + // ... existing tests + RestClientTest.class // Add new test +}) +public class UnitTestSuite {} +``` + +### 4. Refactoring Code + +**Step 1: Ensure Tests Pass** +```bash +# Run all tests before refactoring +mvn test +``` + +**Step 2: Make Changes Incrementally** +```bash +# Small, focused changes +# Run tests after each change +mvn test -Dtest=RelevantTestClass -ntp +``` + +**Step 3: Verify All Tests Still Pass** +```bash +# Run full test suite +mvn test + +# Check code style +mvn checkstyle:check +``` + +**Step 4: Commit** +```bash +git commit -m "refactor(client): extract common HTTP logic to base class + +No functional changes, just code organization improvement." +``` + +## Module-Specific Workflows + +### Working on hugegraph-client + +**Setup Development Environment** +```bash +# Build client only +mvn clean install -pl hugegraph-client -am -DskipTests -ntp + +# Start HugeGraph server for integration tests +# Option 1: Docker +docker run -d --name hugegraph -p 8080:8080 hugegraph/hugegraph + +# Option 2: From source +./hugegraph-client/assembly/travis/install-hugegraph-from-source.sh b7998c1 +``` + +**Development Cycle** +```bash +# 1. Edit code +vim src/main/java/org/apache/hugegraph/api/VertexAPI.java + +# 2. Quick compile check +mvn compile -pl hugegraph-client -ntp + +# 3. Run relevant tests +mvn test -Dtest=VertexApiTest -ntp + +# 4. Full test suite (before commit) +mvn test -Dtest=UnitTestSuite -ntp +mvn test -Dtest=ApiTestSuite +``` + +### Working on hugegraph-loader + +**Setup Development Environment** +```bash +# Build loader with dependencies +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Setup test environment +cd hugegraph-loader/assembly/travis +./install-hugegraph-from-source.sh 5b3d295 + +# For JDBC tests +./install-mysql.sh load_test root + +# For HDFS tests +./install-hadoop.sh +``` + +**Testing New Data Source** +```bash +# 1. Create test data files +mkdir -p src/test/resources/my-test +echo "id,name,age" > src/test/resources/my-test/data.csv +echo "1,Alice,30" >> src/test/resources/my-test/data.csv + +# 2. Create mapping config +vim src/test/resources/struct/my-test.json + +# 3. Write test +vim src/test/java/org/apache/hugegraph/loader/test/functional/MySourceTest.java + +# 4. Run test +mvn test -Dtest=MySourceTest -ntp +``` + +### Working on hugegraph-hubble + +**Setup Development Environment** +```bash +# Ensure Node.js 18.20.8 +node -v # Must be 18.20.8 + +# Install dependencies +npm install -g yarn +cd hugegraph-hubble/hubble-fe +yarn install + +# Install Python requirements (for build) +pip install -r ../hubble-dist/assembly/travis/requirements.txt +``` + +**Frontend Development Cycle** +```bash +cd hugegraph-hubble/hubble-fe + +# 1. Edit code +vim src/components/GraphViewer.tsx + +# 2. Run linter +yarn lint + +# 3. Auto-fix formatting +npx prettier --write src/components/GraphViewer.tsx + +# 4. Run tests +yarn test GraphViewer.test.tsx + +# 5. Start dev server (optional) +yarn start +``` + +**Backend Development Cycle** +```bash +cd hugegraph-hubble/hubble-be + +# 1. Edit code +vim src/main/java/org/apache/hugegraph/hubble/controller/GraphController.java + +# 2. Run tests +mvn test -Dtest=GraphControllerTest -ntp + +# 3. Build and run +mvn spring-boot:run +``` + +**Full Hubble Build** +```bash +# Build dependencies first +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Build hubble +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -ntp + +# Start hubble +cd apache-hugegraph-hubble-incubating-*/ +bin/start-hubble.sh -d + +# Check logs +tail -f logs/hugegraph-hubble.log + +# Access UI +open http://localhost:8088 +``` + +### Working on hugegraph-client-go + +**Setup Development Environment** +```bash +cd hugegraph-client-go + +# Download dependencies +make prepare + +# Setup Go environment +go env -w GO111MODULE=on +``` + +**Development Cycle** +```bash +# 1. Edit code +vim client.go + +# 2. Format code +go fmt ./... + +# 3. Vet code +go vet ./... + +# 4. Run tests with race detector +make test + +# 5. Build binary +make compile + +# 6. Run binary +./hugegraph-client-go +``` + +## Troubleshooting Common Issues + +### Issue: Maven Build Fails with Dependency Errors + +**Solution 1: Clear Local Cache** +```bash +rm -rf ~/.m2/repository/org/apache/hugegraph +mvn clean install -U +``` + +**Solution 2: Use Stage Repository** +```bash +mvn clean install -P stage +``` + +### Issue: Tests Fail with "Connection Refused" + +**Problem**: HugeGraph server not running + +**Solution**: +```bash +# Check if server is running +curl http://localhost:8080/versions + +# If not, start it +cd apache-hugegraph-* +bin/start-hugegraph.sh + +# Wait for startup (check logs) +tail -f logs/hugegraph-server.log +``` + +### Issue: Checkstyle Violations + +**Common Fixes**: +```bash +# Line too long (max 100 chars) +# Solution: Break into multiple lines + +# Star imports +# Solution: Expand imports in IDE (IntelliJ: Ctrl+Alt+O) + +# Wrong indentation +# Solution: Use 4 spaces, not tabs +# IntelliJ: Settings → Editor → Code Style → Java → Indent: 4 + +# Missing whitespace +# Solution: Add space around operators +# Before: if(x==5) +# After: if (x == 5) +``` + +### Issue: Frontend Build Fails + +**Solution 1: Node.js Version** +```bash +# Check version +node -v + +# If wrong version, use nvm +nvm install 18.20.8 +nvm use 18.20.8 +``` + +**Solution 2: Clear Cache** +```bash +cd hugegraph-hubble/hubble-fe +rm -rf node_modules yarn.lock +yarn install +``` + +**Solution 3: Memory Limit** +```bash +# Increase Node.js memory +export NODE_OPTIONS="--max-old-space-size=4096" +mvn clean package +``` + +### Issue: HDFS Tests Fail + +**Solution**: Check Hadoop setup +```bash +# Verify Hadoop is running +jps | grep -E 'NameNode|DataNode' + +# Check HDFS status +hadoop fs -ls / + +# If issues, reinstall +./assembly/travis/install-hadoop.sh +``` + +### Issue: JDBC Tests Fail + +**Solution**: Check MySQL +```bash +# Check MySQL is running +mysql -u root -proot -e "SHOW DATABASES;" + +# Verify test database exists +mysql -u root -proot -e "USE load_test; SHOW TABLES;" + +# If issues, reinstall +./assembly/travis/install-mysql.sh load_test root +``` + +## Release Workflow + +### Preparing a Release + +**Step 1: Update Version** +```bash +# Update root pom.xml +vim pom.xml +# Change 1.7.0 to 1.8.0 + +# Update frontend version +vim hugegraph-hubble/hubble-fe/package.json +# Change "version": "1.7.0" to "version": "1.8.0" +``` + +**Step 2: Update CHANGELOG** +```bash +vim CHANGELOG.md +# Add release notes: +# ## [1.8.0] - 2025-02-01 +# ### Added +# - New batch query API +# ### Fixed +# - CSV loading bug +``` + +**Step 3: Run Full Test Suite** +```bash +# Run all tests +mvn clean verify + +# Run integration tests +cd hugegraph-client && mvn test -Dtest=ApiTestSuite +cd hugegraph-loader && mvn test -P file,hdfs,jdbc +``` + +**Step 4: Build Release Artifacts** +```bash +# Build with Apache release profile +mvn clean package -P apache-release -DskipTests + +# Artifacts in hugegraph-dist/target/ +ls hugegraph-dist/target/*.tar.gz +``` + +**Step 5: Create Release Tag** +```bash +git tag -a v1.8.0 -m "Release version 1.8.0" +git push origin v1.8.0 +``` + +## Useful Development Commands + +### Quick Checks +```bash +# Check what you've changed +git --no-pager diff +git --no-pager diff --staged + +# Check recent commits +git --no-pager log --oneline -5 + +# Find files by name +find . -name "*Test.java" -type f + +# Search in code +grep -r "RestClient" --include="*.java" . +``` + +### Clean Everything +```bash +# Clean Maven build +mvn clean + +# Deep clean +find . -name target -type d -exec rm -rf {} + +find . -name .flattened-pom.xml -delete + +# Clean frontend +cd hugegraph-hubble/hubble-fe +rm -rf node_modules build + +# Clean Go +cd hugegraph-client-go +make clean +``` + +### Performance Profiling +```bash +# Maven build with timing +mvn clean install -Dorg.slf4j.simpleLogger.showDateTime=true + +# Java heap dump on OutOfMemoryError +export MAVEN_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" + +# Go benchmarks +cd hugegraph-client-go +go test -bench=. -benchmem +``` + +## Git Hooks + +### Pre-commit Hook (Optional) +```bash +vim .git/hooks/pre-commit +``` + +```bash +#!/bin/bash +# Pre-commit hook for HugeGraph Toolchain + +# Run checkstyle +echo "Running checkstyle..." +mvn checkstyle:check -q +if [ $? -ne 0 ]; then + echo "Checkstyle failed. Please fix violations." + exit 1 +fi + +# Run license check +echo "Checking licenses..." +mvn apache-rat:check -q +if [ $? -ne 0 ]; then + echo "License check failed. Please add Apache 2.0 headers." + exit 1 +fi + +echo "Pre-commit checks passed." +exit 0 +``` + +```bash +chmod +x .git/hooks/pre-commit +``` + +## IDE Configuration + +### IntelliJ IDEA Setup + +**Import Project**: +1. File → Open → Select `pom.xml` +2. Import as Maven project +3. Wait for dependency resolution + +**Configure Code Style**: +1. Settings → Editor → Code Style → Java +2. Import Scheme → IntelliJ IDEA code style XML +3. Load from: `.editorconfig` + +**Configure Checkstyle Plugin**: +1. Install Checkstyle-IDEA plugin +2. Settings → Tools → Checkstyle +3. Add configuration file: `tools/checkstyle.xml` + +**Run Configurations**: +```xml + + + + + + + + + +``` + +### VS Code Setup + +**Extensions**: +- Java Extension Pack +- Prettier (for Hubble frontend) +- ESLint +- Go (for client-go) + +**Settings** (`.vscode/settings.json`): +```json +{ + "java.configuration.updateBuildConfiguration": "automatic", + "editor.formatOnSave": true, + "editor.tabSize": 4, + "editor.insertSpaces": true, + "[typescript]": { + "editor.tabSize": 2, + "editor.defaultFormatter": "esbenp.prettier-vscode" + }, + "[javascript]": { + "editor.tabSize": 2, + "editor.defaultFormatter": "esbenp.prettier-vscode" + } +} +``` + +## Continuous Learning + +### Understanding the Codebase + +**Start Here**: +1. Read module READMEs +2. Check `example/` directories for usage examples +3. Read test cases to understand expected behavior +4. Follow imports to understand dependencies + +**Key Files to Understand**: +- `hugegraph-client/src/main/java/org/apache/hugegraph/driver/HugeClient.java` +- `hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java` +- `hugegraph-hubble/hubble-fe/src/stores/` (MobX stores) +- `hugegraph-client-go/client.go` + +### Documentation Resources +- Project Docs: https://hugegraph.apache.org/docs/ +- API Docs: https://hugegraph.apache.org/docs/clients/restful-api/ +- GitHub Issues: https://github.com/apache/hugegraph-toolchain/issues +- Mailing List: dev@hugegraph.apache.org diff --git a/.serena/memories/project_overview.md b/.serena/memories/project_overview.md new file mode 100644 index 000000000..4ebec712b --- /dev/null +++ b/.serena/memories/project_overview.md @@ -0,0 +1,126 @@ +# HugeGraph Toolchain Project Overview + +## Project Purpose +Apache HugeGraph Toolchain is an integration project containing a series of utilities for [Apache HugeGraph](https://github.com/apache/hugegraph), a distributed graph database. The toolchain provides essential tools for data loading, management, visualization, and client access. + +## Version Information +- Current Version: 1.7.0 +- License: Apache 2.0 +- Repository: https://github.com/apache/hugegraph-toolchain +- Project Status: Apache Incubator + +## Main Modules (6 Total) + +### 1. hugegraph-client (Java) +**Purpose**: Java RESTful API client for HugeGraph +**Language**: Java 8 +**Key Features**: +- RESTful APIs for accessing graph vertex/edge/schema operations +- Gremlin query support +- Graph traversal algorithms (shortest path, k-hop, etc.) +- Authentication and authorization support + +### 2. hugegraph-loader +**Purpose**: Data loading utility from multiple sources into HugeGraph +**Language**: Java 8 +**Supported Sources**: +- File sources: CSV, JSON, TXT (local files) +- HDFS sources +- JDBC sources: MySQL, PostgreSQL, Oracle, SQL Server +- Kafka streaming sources + +### 3. hugegraph-hubble +**Purpose**: Web-based graph management and analysis dashboard +**Tech Stack**: +- **Backend**: Spring Boot (Java 8) +- **Frontend**: React + TypeScript + MobX (Node.js 18.20.8 required) +**Features**: +- Data loading interface +- Schema management +- Graph visualization +- Query builder (Gremlin and algorithm-based) + +### 4. hugegraph-tools +**Purpose**: Command-line tools for deployment and management +**Language**: Java 8 +**Features**: +- Deployment management +- Backup and restore operations +- Administrative tasks + +### 5. hugegraph-client-go (WIP) +**Purpose**: Go client library for HugeGraph +**Language**: Go +**Status**: Work In Progress + +### 6. hugegraph-spark-connector +**Purpose**: Spark connector for HugeGraph data I/O +**Language**: Java 8 + Scala 2.12 +**Spark Version**: 3.2.2 + +## Module Dependencies +``` +hugegraph-dist (assembly) + └── hugegraph-hubble + └── hugegraph-loader + └── hugegraph-client + └── hugegraph-common (external: v1.5.0) + +hugegraph-tools + └── hugegraph-client + +hugegraph-spark-connector + └── hugegraph-client + +hugegraph-client-go (independent) +``` + +## Technology Stack + +### Java Ecosystem +- **Java Version**: 1.8 (source/target) +- **Build Tool**: Maven 3.x +- **Test Framework**: JUnit 4 + Mockito 2.25.1 +- **Common Libraries**: + - Apache Commons (IO, Lang3, Compress, CLI, Text, Codec) + - Jackson 2.12.3 (JSON processing) + - Log4j2 2.18.0 (Logging) + - Netty 4.1.65.Final + - Lombok 1.18.8 + +### Frontend (Hubble) +- **Node.js**: 18.20.8 (required exact version) +- **Package Manager**: yarn (not npm) +- **Framework**: React +- **Language**: TypeScript +- **State Management**: MobX +- **Code Quality**: Prettier + Stylelint + Husky + +### Go (Client-Go) +- **Build Tool**: Makefile +- **Testing**: Built-in Go test with race detector + +## Key External Dependencies +- HugeGraph Server (required for testing) +- HugeGraph Common library v1.5.0 +- Spark 3.2.2 (for connector) +- Flink 1.13.5 (for stream processing) + +## Project Structure +``` +toolchain/ +├── hugegraph-client/ # Java client library +├── hugegraph-loader/ # Data loading tool +├── hugegraph-hubble/ # Web dashboard +│ ├── hubble-be/ # Backend (Spring Boot) +│ ├── hubble-fe/ # Frontend (React) +│ └── hubble-dist/ # Distribution files +├── hugegraph-tools/ # CLI tools +├── hugegraph-client-go/ # Go client (WIP) +├── hugegraph-spark-connector/# Spark connector +├── hugegraph-dist/ # Assembly module +├── assembly/ # Build descriptors +├── tools/ # Checkstyle, suppressions +├── .github/workflows/ # CI/CD pipelines +└── pom.xml # Root Maven config +``` diff --git a/.serena/memories/task_completion_checklist.md b/.serena/memories/task_completion_checklist.md new file mode 100644 index 000000000..6bb9f2858 --- /dev/null +++ b/.serena/memories/task_completion_checklist.md @@ -0,0 +1,373 @@ +# Task Completion Checklist for HugeGraph Toolchain + +## Before Committing Code + +### 1. Code Quality Checks + +#### Java Modules (client, loader, tools, hubble-be, spark-connector) + +**A. Checkstyle Validation** +```bash +# Run checkstyle on affected modules +mvn checkstyle:check + +# Or for specific module +mvn checkstyle:check -pl hugegraph-client +``` +**Must Pass**: No checkstyle violations allowed + +**B. License Header Check** +```bash +# Verify all files have Apache 2.0 license headers +mvn apache-rat:check +``` +**Must Pass**: All source files must have proper license headers + +**C. EditorConfig Validation** +```bash +# Verify file formatting (indentation, line endings, etc.) +mvn editorconfig:check +``` +**Must Pass**: All files must conform to .editorconfig rules + +**D. Compilation** +```bash +# Ensure code compiles without errors +mvn clean compile -pl -am -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: No compilation errors + +#### Frontend Module (hubble-fe) + +**A. Prettier Formatting** +```bash +cd hugegraph-hubble/hubble-fe + +# Check formatting +npx prettier --check . + +# Auto-fix if needed +npx prettier --write . +``` +**Must Pass**: All files properly formatted + +**B. Stylelint (CSS/Less)** +```bash +# Check CSS/Less files +npx stylelint "**/*.{css,less}" + +# Auto-fix if needed +npx stylelint "**/*.{css,less}" --fix +``` +**Must Pass**: No linting errors + +**C. TypeScript/JavaScript Linting** +```bash +# Run yarn lint +yarn lint +``` +**Must Pass**: No linting errors + +#### Go Module (client-go) + +**A. Go Formatting** +```bash +cd hugegraph-client-go + +# Format code +go fmt ./... + +# Vet code +go vet ./... +``` +**Must Pass**: No formatting or vet issues + +### 2. Run Tests + +#### Java Tests + +**A. Unit Tests** (Always run) +```bash +# For hugegraph-client +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# For hugegraph-loader +cd hugegraph-loader +mvn test -P unit -ntp +``` +**Must Pass**: All unit tests passing + +**B. Integration/API Tests** (If API changes made) +```bash +# For hugegraph-client (requires HugeGraph server) +mvn test -Dtest=ApiTestSuite +mvn test -Dtest=FuncTestSuite + +# For hugegraph-loader (requires HugeGraph server + data sources) +mvn test -P file +mvn test -P hdfs # If HDFS changes +mvn test -P jdbc # If JDBC changes +mvn test -P kafka # If Kafka changes +``` +**Required**: If you modified API/integration code + +#### Frontend Tests +```bash +cd hugegraph-hubble/hubble-fe +yarn test +``` +**Must Pass**: All frontend tests passing + +#### Go Tests +```bash +cd hugegraph-client-go +make test # Runs with race detector +``` +**Must Pass**: All tests passing with no race conditions + +### 3. Build Verification + +#### Full Module Build +```bash +# Build the module(s) you changed +mvn clean install -pl -am -DskipTests -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: Build succeeds without errors + +#### Hubble Build (if frontend/backend changed) +```bash +# Build dependencies +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Build hubble +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: Hubble builds successfully + +### 4. Documentation Updates + +**Check if any of these need updating:** +- [ ] Module README.md +- [ ] API documentation (JavaDoc) +- [ ] Code comments +- [ ] CHANGELOG (if applicable) +- [ ] Version numbers (if release) + +### 5. Git Pre-commit + +**A. Verify Changes** +```bash +# Check what you're committing +git status +git --no-pager diff --staged +``` + +**B. Ensure Proper Commit Message** +Format: `type(scope): subject` + +Examples: +- `feat(client): add batch query support for vertices` +- `fix(loader): handle null values in CSV parsing` +- `docs(hubble): update installation instructions` +- `chore(deps): upgrade jackson to 2.12.3` +- `refactor(tools): simplify backup command logic` +- `test(loader): add HDFS connection retry tests` + +**C. Verify No Unintended Files** +```bash +# Check .gitignore is working +git status --ignored +``` +Do NOT commit: +- `target/` directories +- `*.iml`, `.idea/` files (IDE specific) +- `node_modules/` +- `.flattened-pom.xml` +- Log files +- Build artifacts + +## Pull Request Checklist + +Before opening a PR: + +- [ ] All tests passing locally +- [ ] Code style checks passing (checkstyle, prettier, etc.) +- [ ] No merge conflicts with target branch +- [ ] PR description clearly explains: + - What changed + - Why it changed + - How to test it +- [ ] Reference issue number (if applicable): `Fixes #123` +- [ ] Updated documentation (if applicable) +- [ ] Added tests for new functionality +- [ ] CI builds passing on GitHub Actions + +## CI/CD Pipeline Checks + +The following will be automatically checked by GitHub Actions: + +### Java Client CI (`client-ci.yml`) +1. Compile hugegraph-client +2. Run UnitTestSuite +3. Run ApiTestSuite (requires HugeGraph server) +4. Run FuncTestSuite +5. Upload coverage to Codecov + +### Loader CI (`loader-ci.yml`) +1. Install dependencies (Hadoop, MySQL, HugeGraph) +2. Compile client + loader +3. Run unit tests (`-P unit`) +4. Run file tests (`-P file`) +5. Run HDFS tests (`-P hdfs`) +6. Run JDBC tests (`-P jdbc`) +7. Run Kafka tests (`-P kafka`) +8. Upload coverage to Codecov + +### Hubble CI (`hubble-ci.yml`) +1. Setup Node.js 18.20.8 +2. Install frontend dependencies (yarn) +3. Build frontend (React + TypeScript) +4. Compile backend (Spring Boot) +5. Run tests +6. Package distribution + +### Go Client CI (`client-go-ci.yml`) +1. Setup Go environment +2. Download dependencies +3. Run `make test` (with race detector) +4. Build binary + +### Tools CI (`tools-ci.yml`) +1. Compile hugegraph-tools +2. Run tests +3. Package distribution + +### Spark Connector CI (`spark-connector-ci.yml`) +1. Setup Scala environment +2. Compile spark-connector +3. Run tests + +### CodeQL Analysis (`codeql-analysis.yml`) +- Security vulnerability scanning +- Code quality analysis + +### License Checker (`license-checker.yml`) +- Verify Apache 2.0 license headers +- Check dependency licenses + +## Common Issues and Solutions + +### Issue: Checkstyle Failures +**Solution**: +1. Check error message for specific rule violation +2. Fix manually or use IDE auto-format (IntelliJ IDEA) +3. Common issues: + - Line too long (max 100 chars) + - Star imports + - Missing whitespace + - Wrong indentation (use 4 spaces) + +### Issue: Test Failures +**Solution**: +1. Check if HugeGraph server is running (for API/Func tests) +2. Verify dependencies are installed (HDFS, MySQL, Kafka) +3. Check test logs for specific error +4. Run single test for debugging: + ```bash + mvn test -Dtest=ClassName#methodName -ntp + ``` + +### Issue: Hubble Build Failures +**Solution**: +1. Verify Node.js version: `node -v` (must be 18.20.8) +2. Clear cache and reinstall: + ```bash + rm -rf node_modules yarn.lock + yarn install + ``` +3. Check for frontend errors in build output + +### Issue: Maven Build Hangs +**Solution**: +1. Kill stuck maven process: `pkill -9 -f maven` +2. Clear local repository cache: + ```bash + rm -rf ~/.m2/repository/org/apache/hugegraph + ``` +3. Retry with `-X` for debug output: + ```bash + mvn clean install -X + ``` + +## Release-Specific Tasks + +When preparing a release: + +1. **Update Version Numbers** + - Root `pom.xml`: `` property + - Frontend: `package.json` version + - Go: Version constants + +2. **Update CHANGELOG** + - Document new features + - List bug fixes + - Note breaking changes + +3. **Run Full Test Suite** + ```bash + mvn clean verify -P apache-release + ``` + +4. **Generate Distribution** + ```bash + mvn clean package -DskipTests + ``` + +5. **Sign Artifacts** (for Apache release) + ```bash + mvn clean install -P apache-release + ``` + +## Summary - Minimum Required Checks + +**For any code change, ALWAYS run:** + +```bash +# 1. Checkstyle (Java) +mvn checkstyle:check + +# 2. License check +mvn apache-rat:check + +# 3. EditorConfig +mvn editorconfig:check + +# 4. Unit tests +mvn test -Dtest=UnitTestSuite -ntp # or appropriate suite + +# 5. Build +mvn clean install -DskipTests -ntp +``` + +**For frontend changes, ALSO run:** +```bash +cd hugegraph-hubble/hubble-fe +npx prettier --check . +npx stylelint "**/*.{css,less}" +yarn lint +yarn test +``` + +**For Go changes, ALSO run:** +```bash +cd hugegraph-client-go +go fmt ./... +go vet ./... +make test +``` + +--- + +**CRITICAL**: Do NOT commit code that fails any of the required checks. CI will fail and PR will be blocked. diff --git a/.serena/memories/testing_infrastructure.md b/.serena/memories/testing_infrastructure.md new file mode 100644 index 000000000..66f785c80 --- /dev/null +++ b/.serena/memories/testing_infrastructure.md @@ -0,0 +1,635 @@ +# Testing Infrastructure - HugeGraph Toolchain + +## Testing Philosophy + +- **Unit Tests**: Test individual components in isolation, no external dependencies +- **Integration Tests**: Test interactions with HugeGraph server and external systems +- **Functional Tests**: End-to-end workflows testing complete features + +## Test Organization + +### Test Suite Structure (Java Modules) + +All Java modules use **JUnit 4** with test suites: + +``` +src/test/java/ +├── unit/ +│ ├── *Test.java # Individual unit tests +│ └── UnitTestSuite.java # Suite aggregator +├── api/ +│ ├── *ApiTest.java # API integration tests +│ └── ApiTestSuite.java +└── functional/ + ├── *FuncTest.java # Functional tests + └── FuncTestSuite.java +``` + +### Test Naming Conventions + +**Class Names**: +- Unit tests: `ClassNameTest.java` +- Integration tests: `ClassNameApiTest.java` or `ClassNameIntegrationTest.java` +- Test suites: `UnitTestSuite.java`, `ApiTestSuite.java`, `FuncTestSuite.java` + +**Method Names**: +- Descriptive: `testGetVertexById()`, `testCreateEdgeWithInvalidLabel()` +- Pattern: `test()` + +## Module-Specific Testing + +## 1. hugegraph-client Tests + +### Test Suites + +#### UnitTestSuite +**Purpose**: Test serialization, utilities, and internal logic +**No External Dependencies**: Can run without HugeGraph server + +**Example Tests**: +```java +@RunWith(Suite.class) +@Suite.SuiteClasses({ + VertexSerializerTest.class, + PathSerializerTest.class, + RestResultTest.class, + BatchElementRequestTest.class, + PropertyKeyTest.class, + IndexLabelTest.class, + CommonUtilTest.class, + IdUtilTest.class, + SplicingIdGeneratorTest.class +}) +public class UnitTestSuite {} +``` + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp +``` + +#### ApiTestSuite +**Purpose**: Test REST API interactions +**Requires**: HugeGraph server running on localhost:8080 + +**Example Tests**: +- `VertexApiTest`: Test vertex CRUD operations +- `EdgeApiTest`: Test edge CRUD operations +- `SchemaApiTest`: Test schema management +- `TraverserApiTest`: Test graph traversal algorithms +- `GremlinApiTest`: Test Gremlin query execution + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=ApiTestSuite +``` + +#### FuncTestSuite +**Purpose**: End-to-end functional scenarios +**Requires**: HugeGraph server + complete setup + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=FuncTestSuite +``` + +### Test Setup/Teardown Pattern + +```java +public class VertexApiTest extends BaseApiTest { + private static HugeClient client; + private static GraphManager graph; + + @BeforeClass + public static void setup() { + client = new HugeClient("http://localhost:8080", "hugegraph"); + graph = client.graph(); + + // Setup schema + setupSchema(); + } + + @AfterClass + public static void teardown() { + client.close(); + } + + @Before + public void prepare() { + // Clear data before each test + graph.clearVertices(); + } + + @Test + public void testAddVertex() { + Vertex vertex = graph.addVertex("person", "name", "Alice"); + assertNotNull(vertex.id()); + assertEquals("Alice", vertex.property("name")); + } +} +``` + +### Mocking (Unit Tests) + +```java +public class RestClientTest { + @Mock + private RestClient mockClient; + + @Before + public void setup() { + MockitoAnnotations.initMocks(this); + } + + @Test + public void testGetVertex() { + // Mock response + Vertex expectedVertex = new Vertex("person"); + when(mockClient.get("/vertices/1", Vertex.class)) + .thenReturn(expectedVertex); + + // Test + Vertex result = mockClient.get("/vertices/1", Vertex.class); + assertEquals(expectedVertex, result); + + // Verify + verify(mockClient).get("/vertices/1", Vertex.class); + } +} +``` + +## 2. hugegraph-loader Tests + +### Test Profiles (Maven) + +#### Profile: unit +**Purpose**: Unit tests only +**Run Command**: +```bash +cd hugegraph-loader +mvn test -P unit -ntp +``` + +**Tests**: Parser, mapper, builder unit tests + +#### Profile: file +**Purpose**: File source loading tests +**Requires**: Test data files (CSV, JSON, TXT) +**Run Command**: +```bash +mvn test -P file +``` + +**Test Resources**: +``` +src/test/resources/ +├── file/ +│ ├── persons.csv +│ ├── knows.json +│ └── struct.json # Mapping configuration +``` + +#### Profile: hdfs +**Purpose**: HDFS source loading tests +**Requires**: Hadoop HDFS cluster (local or remote) +**Setup**: CI installs Hadoop via `install-hadoop.sh` +**Run Command**: +```bash +mvn test -P hdfs +``` + +#### Profile: jdbc +**Purpose**: Database source loading tests +**Requires**: MySQL running (CI uses Docker) +**Setup**: CI installs MySQL via `install-mysql.sh` +**Run Command**: +```bash +mvn test -P jdbc +``` + +**Test Databases**: MySQL, PostgreSQL, Oracle (if driver available) + +#### Profile: kafka +**Purpose**: Kafka streaming source tests +**Requires**: Kafka broker running +**Run Command**: +```bash +mvn test -P kafka +``` + +### Test Data Management + +**Test Resources Structure**: +``` +src/test/resources/ +├── struct/ +│ ├── vertices.json # Vertex mapping configs +│ └── edges.json # Edge mapping configs +├── file/ +│ ├── vertex_person.csv +│ ├── edge_knows.csv +│ └── example.json +├── jdbc/ +│ └── init.sql # Database init script +└── log4j2.xml # Test logging config +``` + +### Integration Test Pattern (Loader) + +```java +public class FileLoadTest extends BaseLoadTest { + private static LoadContext context; + private static HugeClient client; + + @BeforeClass + public static void setup() { + // Start HugeGraph server (CI does this) + client = new HugeClient("http://localhost:8080", "hugegraph"); + + // Create schema + createSchema(client); + + // Prepare load context + context = new LoadContext(); + context.setStructPath("src/test/resources/struct/vertices.json"); + } + + @Test + public void testLoadCSV() { + // Load data + LoadOptions options = new LoadOptions(); + options.file = "src/test/resources/file/vertex_person.csv"; + + HugeGraphLoader loader = new HugeGraphLoader(context, options); + loader.load(); + + // Verify + List vertices = client.graph().listVertices("person"); + assertEquals(100, vertices.size()); + } +} +``` + +## 3. hugegraph-hubble Tests + +### Backend Tests (Java/Spring Boot) + +**Test Framework**: JUnit 4 + Spring Test + MockMvc + +**Example Controller Test**: +```java +@RunWith(SpringRunner.class) +@WebMvcTest(GraphConnectionController.class) +public class GraphConnectionControllerTest { + @Autowired + private MockMvc mockMvc; + + @MockBean + private GraphConnectionService service; + + @Test + public void testCreateConnection() throws Exception { + GraphConnection connection = new GraphConnection(); + connection.setName("test-graph"); + connection.setHost("localhost"); + connection.setPort(8080); + + when(service.create(any())).thenReturn(connection); + + mockMvc.perform(post("/api/graph-connections") + .contentType(MediaType.APPLICATION_JSON) + .content(toJson(connection))) + .andExpect(status().isOk()) + .andExpect(jsonPath("$.name").value("test-graph")); + } +} +``` + +### Frontend Tests (React/TypeScript) + +**Test Framework**: Jest + React Testing Library + +**Run Command**: +```bash +cd hugegraph-hubble/hubble-fe +yarn test +``` + +**Example Component Test**: +```typescript +import { render, screen, fireEvent } from '@testing-library/react'; +import GraphSelector from '../GraphSelector'; + +describe('GraphSelector', () => { + it('renders graph list', () => { + const graphs = [ + { id: 1, name: 'graph1' }, + { id: 2, name: 'graph2' } + ]; + + render(); + + expect(screen.getByText('graph1')).toBeInTheDocument(); + expect(screen.getByText('graph2')).toBeInTheDocument(); + }); + + it('calls onSelect when graph clicked', () => { + const onSelect = jest.fn(); + const graphs = [{ id: 1, name: 'graph1' }]; + + render(); + + fireEvent.click(screen.getByText('graph1')); + expect(onSelect).toHaveBeenCalledWith(graphs[0]); + }); +}); +``` + +**Store Test (MobX)**: +```typescript +import GraphManagementStore from '../stores/GraphManagementStore'; + +describe('GraphManagementStore', () => { + let store: GraphManagementStore; + + beforeEach(() => { + store = new GraphManagementStore(); + }); + + it('loads graphs from API', async () => { + // Mock API + jest.spyOn(api, 'getGraphs').mockResolvedValue([ + { id: 1, name: 'graph1' } + ]); + + await store.loadGraphs(); + + expect(store.graphList).toHaveLength(1); + expect(store.graphList[0].name).toBe('graph1'); + }); +}); +``` + +## 4. hugegraph-client-go Tests + +**Test Framework**: Go standard testing + testify + +**Run Command**: +```bash +cd hugegraph-client-go +make test # Runs: go test -race -timeout 30s +``` + +**Test Structure**: +``` +. +├── client_test.go +├── graph_test.go +├── schema_test.go +└── traverser_test.go +``` + +**Example Test**: +```go +package hugegraph + +import ( + "testing" + "github.com/stretchr/testify/assert" +) + +func TestCreateVertex(t *testing.T) { + client := NewClient("http://localhost:8080", "hugegraph") + defer client.Close() + + vertex := Vertex{ + Label: "person", + Properties: map[string]interface{}{ + "name": "Alice", + "age": 30, + }, + } + + created, err := client.Graph().AddVertex(vertex) + assert.NoError(t, err) + assert.NotEmpty(t, created.ID) + assert.Equal(t, "Alice", created.Properties["name"]) +} + +func TestGetVertexNotFound(t *testing.T) { + client := NewClient("http://localhost:8080", "hugegraph") + defer client.Close() + + _, err := client.Graph().GetVertex("non-existent-id") + assert.Error(t, err) +} +``` + +**Benchmark Tests**: +```go +func BenchmarkAddVertex(b *testing.B) { + client := NewClient("http://localhost:8080", "hugegraph") + defer client.Close() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + client.Graph().AddVertex(Vertex{ + Label: "person", + Properties: map[string]interface{}{"name": "test"}, + }) + } +} +``` + +## CI/CD Testing Pipeline + +### GitHub Actions Workflow + +Each module has its own CI workflow: + +#### client-ci.yml +```yaml +steps: + - name: Install HugeGraph Server + run: ./assembly/travis/install-hugegraph-from-source.sh + + - name: Compile + run: mvn compile -pl hugegraph-client -ntp + + - name: Run Unit Tests + run: mvn test -Dtest=UnitTestSuite -ntp + + - name: Run API Tests + run: mvn test -Dtest=ApiTestSuite + + - name: Run Func Tests + run: mvn test -Dtest=FuncTestSuite + + - name: Upload Coverage + uses: codecov/codecov-action@v3 +``` + +#### loader-ci.yml +```yaml +steps: + - name: Install Dependencies + run: | + ./assembly/travis/install-hadoop.sh + ./assembly/travis/install-mysql.sh + ./assembly/travis/install-hugegraph-from-source.sh + + - name: Run Tests + run: | + mvn test -P unit + mvn test -P file + mvn test -P hdfs + mvn test -P jdbc + mvn test -P kafka +``` + +### Test Utilities + +#### CI Setup Scripts +```bash +# Install HugeGraph server from source +./assembly/travis/install-hugegraph-from-source.sh + +# Install Hadoop for HDFS tests +./assembly/travis/install-hadoop.sh + +# Install MySQL for JDBC tests +./assembly/travis/install-mysql.sh +``` + +## Test Coverage + +### Coverage Tools +- **Java**: JaCoCo Maven plugin +- **JavaScript/TypeScript**: Jest built-in coverage +- **Go**: go test -cover + +### Generating Coverage Reports + +**Java (JaCoCo)**: +```bash +mvn test jacoco:report +# Report: target/site/jacoco/index.html +``` + +**Frontend (Jest)**: +```bash +cd hugegraph-hubble/hubble-fe +yarn test --coverage +# Report: coverage/lcov-report/index.html +``` + +**Go**: +```bash +cd hugegraph-client-go +go test -coverprofile=coverage.out ./... +go tool cover -html=coverage.out +``` + +### Coverage Targets +- Unit tests: Aim for 80%+ coverage +- Integration tests: Cover critical paths +- Functional tests: Cover end-to-end scenarios + +## Common Testing Patterns + +### Test Data Builders +```java +public class TestDataBuilder { + public static Vertex createPersonVertex(String name, int age) { + return new Vertex("person") + .property("name", name) + .property("age", age); + } + + public static Edge createKnowsEdge(Vertex source, Vertex target) { + return source.addEdge("knows", target) + .property("date", "2023-01-01"); + } +} +``` + +### Test Assertions (Custom) +```java +public class GraphAssertions { + public static void assertVertexExists(HugeClient client, Object id) { + Vertex vertex = client.graph().getVertex(id); + assertNotNull("Vertex should exist", vertex); + } + + public static void assertEdgeCount(HugeClient client, + String label, int expected) { + List edges = client.graph().listEdges(label); + assertEquals("Edge count mismatch", expected, edges.size()); + } +} +``` + +### Parameterized Tests (JUnit 4) +```java +@RunWith(Parameterized.class) +public class IdGeneratorTest { + @Parameters + public static Collection data() { + return Arrays.asList(new Object[][] { + { "alice", "person:alice" }, + { "bob", "person:bob" }, + { "charlie", "person:charlie" } + }); + } + + private String input; + private String expected; + + public IdGeneratorTest(String input, String expected) { + this.input = input; + this.expected = expected; + } + + @Test + public void testGenerateId() { + String result = IdGenerator.generate("person", input); + assertEquals(expected, result); + } +} +``` + +## Debugging Tests + +### Running Single Test +```bash +# Java +mvn test -Dtest=ClassName#methodName -ntp + +# Go +go test -run TestFunctionName -v + +# Frontend +yarn test ComponentName.test.tsx +``` + +### Debug Mode (Java) +```bash +# Run with remote debugging enabled +mvnDebug test -Dtest=ClassName +# Then attach debugger to port 8000 +``` + +### Verbose Output +```bash +# Maven verbose +mvn test -X + +# Go verbose +go test -v + +# Frontend verbose +yarn test --verbose +``` diff --git a/.serena/project.yml b/.serena/project.yml new file mode 100644 index 000000000..bd75e5b35 --- /dev/null +++ b/.serena/project.yml @@ -0,0 +1,84 @@ +# list of languages for which language servers are started; choose from: +# al bash clojure cpp csharp csharp_omnisharp +# dart elixir elm erlang fortran go +# haskell java julia kotlin lua markdown +# nix perl php python python_jedi r +# rego ruby ruby_solargraph rust scala swift +# terraform typescript typescript_vts zig +# Note: +# - For C, use cpp +# - For JavaScript, use typescript +# Special requirements: +# - csharp: Requires the presence of a .sln file in the project folder. +# When using multiple languages, the first language server that supports a given file will be used for that file. +# The first language is the default language and the respective language server will be used as a fallback. +# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored. +languages: +- java + +# the encoding used by text files in the project +# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings +encoding: "utf-8" + +# whether to use the project's gitignore file to ignore files +# Added on 2025-04-07 +ignore_all_files_in_gitignore: true + +# list of additional paths to ignore +# same syntax as gitignore, so you can use * and ** +# Was previously called `ignored_dirs`, please update your config if you are using that. +# Added (renamed) on 2025-04-07 +ignored_paths: [] + +# whether the project is in read-only mode +# If set to true, all editing tools will be disabled and attempts to use them will result in an error +# Added on 2025-04-18 +read_only: false + +# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details. +# Below is the complete list of tools for convenience. +# To make sure you have the latest list of tools, and to view their descriptions, +# execute `uv run scripts/print_tool_overview.py`. +# +# * `activate_project`: Activates a project by name. +# * `check_onboarding_performed`: Checks whether project onboarding was already performed. +# * `create_text_file`: Creates/overwrites a file in the project directory. +# * `delete_lines`: Deletes a range of lines within a file. +# * `delete_memory`: Deletes a memory from Serena's project-specific memory store. +# * `execute_shell_command`: Executes a shell command. +# * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced. +# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type). +# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type). +# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes. +# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file. +# * `initial_instructions`: Gets the initial instructions for the current project. +# Should only be used in settings where the system prompt cannot be set, +# e.g. in clients you have no control over, like Claude Desktop. +# * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol. +# * `insert_at_line`: Inserts content at a given line in a file. +# * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol. +# * `list_dir`: Lists files and directories in the given directory (optionally with recursion). +# * `list_memories`: Lists memories in Serena's project-specific memory store. +# * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building). +# * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context). +# * `read_file`: Reads a file within the project directory. +# * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store. +# * `remove_project`: Removes a project from the Serena configuration. +# * `replace_lines`: Replaces a range of lines within a file with new content. +# * `replace_symbol_body`: Replaces the full definition of a symbol. +# * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen. +# * `search_for_pattern`: Performs a search for a pattern in the project. +# * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase. +# * `switch_modes`: Activates modes by providing a list of their names +# * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information. +# * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task. +# * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed. +# * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store. +excluded_tools: [] + +# initial prompt for the project. It will always be given to the LLM upon activating the project +# (contrary to the memories, which are loaded on demand). +initial_prompt: "" + +project_name: "toolchain" +included_optional_tools: [] diff --git a/pom.xml b/pom.xml index e9978ff4f..d9413db92 100644 --- a/pom.xml +++ b/pom.xml @@ -493,6 +493,8 @@ **/*.svg .github/**/* + + .serena/**/* **/*.iml **/*.iws