diff --git a/docs/content/Cluster-setup.md b/docs/content/Cluster-setup.md index aa142efc4538..e4ba0e564f11 100644 --- a/docs/content/Cluster-setup.md +++ b/docs/content/Cluster-setup.md @@ -1,6 +1,9 @@ --- layout: doc_page --- + +# Setting Up a Druid Cluster + A Druid cluster consists of various node types that need to be set up depending on your use case. See our [Design](Design.html) docs for a description of the different node types. Minimum Physical Layout: Absolute Minimum @@ -74,7 +77,7 @@ Local disk ("ephemeral" on AWS EC2) for caching is recommended over network moun Setup ----- -Setting up a cluster is essentially just firing up all of the nodes you want with the proper [[configuration]]. One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process: +Setting up a cluster is essentially just firing up all of the nodes you want with the proper [configuration](Configuration.html). One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process: ``` druid.server.type=historical|realtime diff --git a/docs/content/Configuration.md b/docs/content/Configuration.md index 51afb07ae654..47c495f01423 100644 --- a/docs/content/Configuration.md +++ b/docs/content/Configuration.md @@ -1,25 +1,28 @@ --- layout: doc_page --- + +# Configuring Druid + This describes the basic server configuration that is loaded by all the server processes; the same file is loaded by all. See also the json "specFile" descriptions in [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html). -JVM Configuration Best Practices -================================ +## JVM Configuration Best Practices There are three JVM parameters that we set on all of our processes: -1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs -2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs -3. `-Djava.io.tmpdir=` Various parts of the system that interact with the file system do it via temporary files, these files can get somewhat large. Many production systems are setup to have small (but fast) `/tmp` directories, these can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat. +1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs. +2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs. +3. `-Djava.io.tmpdir=` Various parts of the system that interact with the file system do it via temporary files, and these files can get somewhat large. Many production systems are set up to have small (but fast) `/tmp` directories, which can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat. + +## Modules -Modules -======= +As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. -As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules. +This page describes the configuration of the default modules. Node-specific configuration is discussed on each node's respective page. In addition, you can add custom modules to [extend Druid](Modules.html). Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath. -Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration files. +Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration file. ### Emitter Module @@ -147,7 +150,7 @@ Druid storage nodes maintain information about segments they have already downlo |Property|Description|Default| |--------|-----------|-------| -|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations defines where that local cache resides|none| +|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations define where that local cache resides|none| |`druid.segmentCache.deleteOnRemove`|Delete segment files from cache once a node is no longer serving a segment.|true| |`druid.segmentCache.infoDir`|Historical nodes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the coordinator to reassign. This path defines where this metadata is kept. Directory will be created if needed.|${first_location}/info_dir| diff --git a/docs/content/Modules.md b/docs/content/Modules.md index 17b8e5387855..b5b8a693053d 100644 --- a/docs/content/Modules.md +++ b/docs/content/Modules.md @@ -1,6 +1,9 @@ --- layout: doc_page --- + +# Extending Druid With Custom Modules + Druid version 0.6 introduces a new module system that allows for the addition of extensions at runtime. ## Specifying extensions @@ -164,4 +167,4 @@ Adding new Jersey resources to a module requires calling the following code to b ```java Jerseys.addResource(binder, NewResource.class); -``` \ No newline at end of file +``` diff --git a/docs/content/toc.textile b/docs/content/toc.textile index 84fde0369977..ae0da14bcc70 100644 --- a/docs/content/toc.textile +++ b/docs/content/toc.textile @@ -3,7 +3,7 @@ -h1. Introduction +h2. Introduction * "About Druid":./ * "Concepts and Terminology":./Concepts-and-Terminology.html @@ -14,15 +14,11 @@ h2. Getting Started * "Tutorial: Loading Your Data Part 2":./Tutorial:-Loading-Your-Data-Part-2.html * "Tutorial: All About Queries":./Tutorial:-All-About-Queries.html -h2. Evaluate Druid +h2. Operations * "Cluster Setup":./Cluster-setup.html -* "Booting a Production Cluster":./Booting-a-production-cluster.html - -h2. Configuration * "Configuration":Configuration.html - -h2. Extend Druid -* "Modules":./Modules.html +* "Extending Druid":./Modules.html +* "Booting a Production Cluster":./Booting-a-production-cluster.html h2. Data Ingestion * "Realtime":./Realtime.html