Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/content/Cluster-setup.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
---
layout: doc_page
---

# Setting Up a Druid Cluster

A Druid cluster consists of various node types that need to be set up depending on your use case. See our [Design](Design.html) docs for a description of the different node types.

Minimum Physical Layout: Absolute Minimum
Expand Down Expand Up @@ -74,7 +77,7 @@ Local disk ("ephemeral" on AWS EC2) for caching is recommended over network moun
Setup
-----

Setting up a cluster is essentially just firing up all of the nodes you want with the proper [[configuration]]. One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process:
Setting up a cluster is essentially just firing up all of the nodes you want with the proper [configuration](Configuration.html). One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process:

```
druid.server.type=historical|realtime
Expand Down
23 changes: 13 additions & 10 deletions docs/content/Configuration.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,28 @@
---
layout: doc_page
---

# Configuring Druid

This describes the basic server configuration that is loaded by all the server processes; the same file is loaded by all. See also the json "specFile" descriptions in [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html).

JVM Configuration Best Practices
================================
## JVM Configuration Best Practices

There are three JVM parameters that we set on all of our processes:

1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs
2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs
3. `-Djava.io.tmpdir=<a path>` Various parts of the system that interact with the file system do it via temporary files, these files can get somewhat large. Many production systems are setup to have small (but fast) `/tmp` directories, these can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat.
1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs.
2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs.
3. `-Djava.io.tmpdir=<a path>` Various parts of the system that interact with the file system do it via temporary files, and these files can get somewhat large. Many production systems are set up to have small (but fast) `/tmp` directories, which can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat.

## Modules

Modules
=======
As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration.

As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules.
This page describes the configuration of the default modules. Node-specific configuration is discussed on each node's respective page. In addition, you can add custom modules to [extend Druid](Modules.html).

Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath.

Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration files.
Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration file.

### Emitter Module

Expand Down Expand Up @@ -147,7 +150,7 @@ Druid storage nodes maintain information about segments they have already downlo

|Property|Description|Default|
|--------|-----------|-------|
|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations defines where that local cache resides|none|
|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations define where that local cache resides|none|
|`druid.segmentCache.deleteOnRemove`|Delete segment files from cache once a node is no longer serving a segment.|true|
|`druid.segmentCache.infoDir`|Historical nodes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the coordinator to reassign. This path defines where this metadata is kept. Directory will be created if needed.|${first_location}/info_dir|

Expand Down
5 changes: 4 additions & 1 deletion docs/content/Modules.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
---
layout: doc_page
---

# Extending Druid With Custom Modules

Druid version 0.6 introduces a new module system that allows for the addition of extensions at runtime.

## Specifying extensions
Expand Down Expand Up @@ -164,4 +167,4 @@ Adding new Jersey resources to a module requires calling the following code to b

```java
Jerseys.addResource(binder, NewResource.class);
```
```
12 changes: 4 additions & 8 deletions docs/content/toc.textile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

<link rel="stylesheet" href="css/toc.css">

h1. Introduction
h2. Introduction
* "About Druid":./
* "Concepts and Terminology":./Concepts-and-Terminology.html

Expand All @@ -14,15 +14,11 @@ h2. Getting Started
* "Tutorial: Loading Your Data Part 2":./Tutorial:-Loading-Your-Data-Part-2.html
* "Tutorial: All About Queries":./Tutorial:-All-About-Queries.html

h2. Evaluate Druid
h2. Operations
* "Cluster Setup":./Cluster-setup.html
* "Booting a Production Cluster":./Booting-a-production-cluster.html

h2. Configuration
* "Configuration":Configuration.html

h2. Extend Druid
* "Modules":./Modules.html
* "Extending Druid":./Modules.html
* "Booting a Production Cluster":./Booting-a-production-cluster.html

h2. Data Ingestion
* "Realtime":./Realtime.html
Expand Down