Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/development/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Druid's extensions leverage Guice in order to add things at runtime. Basically,
1. Add new Jersey resources by calling `Jerseys.addResource(binder, clazz)`.
1. Add new Jetty filters by extending `org.apache.druid.server.initialization.jetty.ServletFilterHolder`.
1. Add new secret providers by extending `org.apache.druid.metadata.PasswordProvider`.
1. Add new ingest transform by implementing the `org.apache.druid.segment.transform.Transform` interface from the `druid-processing` package.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense to me to add a new item in the list of 15 items under "Writing your own extensions" for completeness sake. I'd be happy to write a dedicated section of these docs, although I think this page seems destined to be split into many pages: one describing the approach for each kind of extension.

Hi @mitchlloyd, I assume you are interested in writing docs what @suneet-s mentioned based on your comment above. Do you want to add them in this PR? I think that could be done in a follow-up PR if you want.

1. Bundle your extension with all the other Druid extensions

Extensions are added to the system via an implementation of `org.apache.druid.initialization.DruidModule`.
Expand Down Expand Up @@ -239,6 +240,72 @@ In your implementation of `org.apache.druid.initialization.DruidModule`, `getJac

where `SomePasswordProvider` is the implementation of `PasswordProvider` interface, you can have a look at `org.apache.druid.metadata.EnvironmentVariablePasswordProvider` for example.

### Adding a Transform Extension

To create a transform extension implement the `org.apache.druid.segment.transform.Transform` interface. You'll need to install the `druid-processing` package to import `org.apache.druid.segment.transform`.

```java
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.apache.druid.segment.transform.RowFunction;
import org.apache.druid.segment.transform.Transform;

public class MyTransform implements Transform {
private final String name;

@JsonCreator
public MyTransform(
@JsonProperty("name") final String name
) {
this.name = name;
}

@JsonProperty
@Override
public String getName() {
return name;
}

@Override
public RowFunction getRowFunction() {
return new MyRowFunction();
}

static class MyRowFunction implements RowFunction {
@Override
public Object eval(Row row) {
return "transformed-value";
}
}
}
```

Then register your transform as a Jackson module.

```java
import com.fasterxml.jackson.databind.Module;
import com.fasterxml.jackson.databind.jsontype.NamedModule;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.google.inject.Binder;
import com.google.common.collect.ImmutableList;
import org.apache.druid.initialization.DruidModule;

public class MyTransformModule implements DruidModule {
@Override
public List<? extends Module> getJacksonModules() {
return return ImmutableList.of(
new SimpleModule("MyTransformModule").registerSubtypes(
new NamedType(MyTransform.class, "my-transform")
)
):
}

@Override
public void configure(Binder binder) {
}
}
```

### Bundle your extension with all the other Druid extensions

When you do `mvn install`, Druid extensions will be packaged within the Druid tarball and `extensions` directory, which are both underneath `distribution/target/`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import org.apache.druid.guice.annotations.ExtensionPoint;

/**
* A row transform that is part of a {@link TransformSpec}. Transforms allow adding new fields to input rows. Each
Expand All @@ -34,6 +35,7 @@
* they cannot refer to other transforms. And they cannot remove fields, only add them. However, they can shadow a
* field with another field containing all nulls, which will act similarly to removing the field.
*/
@ExtensionPoint
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes(value = {
@JsonSubTypes.Type(name = "expression", value = ExpressionTransform.class)
Expand Down