From 84234b142d1c89f6beaf152d0f1d55544d795e45 Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Mon, 9 Dec 2013 13:21:57 -0800 Subject: [PATCH 1/6] added title; minor typo fixes --- docs/content/Configuration.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/content/Configuration.md b/docs/content/Configuration.md index 51afb07ae654..1934b941bc31 100644 --- a/docs/content/Configuration.md +++ b/docs/content/Configuration.md @@ -1,25 +1,26 @@ --- layout: doc_page --- + +# Configuring Druid + This describes the basic server configuration that is loaded by all the server processes; the same file is loaded by all. See also the json "specFile" descriptions in [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html). -JVM Configuration Best Practices -================================ +## JVM Configuration Best Practices There are three JVM parameters that we set on all of our processes: -1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs -2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs -3. `-Djava.io.tmpdir=` Various parts of the system that interact with the file system do it via temporary files, these files can get somewhat large. Many production systems are setup to have small (but fast) `/tmp` directories, these can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat. +1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs. +2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs. +3. `-Djava.io.tmpdir=` Various parts of the system that interact with the file system do it via temporary files, and these files can get somewhat large. Many production systems are set up to have small (but fast) `/tmp` directories, which can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat. -Modules -======= +## Modules As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules. Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath. -Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration files. +Note: as a future item, we’d like to consolidate all of the various configuration into a yaml/JSON based configuration file. ### Emitter Module @@ -147,7 +148,7 @@ Druid storage nodes maintain information about segments they have already downlo |Property|Description|Default| |--------|-----------|-------| -|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations defines where that local cache resides|none| +|`druid.segmentCache.locations`|Segments assigned to a historical node are first stored on the local file system and then served by the historical node. These locations define where that local cache resides|none| |`druid.segmentCache.deleteOnRemove`|Delete segment files from cache once a node is no longer serving a segment.|true| |`druid.segmentCache.infoDir`|Historical nodes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the coordinator to reassign. This path defines where this metadata is kept. Directory will be created if needed.|${first_location}/info_dir| From ddbab46ace9c09453d190b25c8d98ced2bc103e0 Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Mon, 9 Dec 2013 14:03:29 -0800 Subject: [PATCH 2/6] added title; fixed link to configuration --- docs/content/Cluster-setup.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/content/Cluster-setup.md b/docs/content/Cluster-setup.md index aa142efc4538..e4ba0e564f11 100644 --- a/docs/content/Cluster-setup.md +++ b/docs/content/Cluster-setup.md @@ -1,6 +1,9 @@ --- layout: doc_page --- + +# Setting Up a Druid Cluster + A Druid cluster consists of various node types that need to be set up depending on your use case. See our [Design](Design.html) docs for a description of the different node types. Minimum Physical Layout: Absolute Minimum @@ -74,7 +77,7 @@ Local disk ("ephemeral" on AWS EC2) for caching is recommended over network moun Setup ----- -Setting up a cluster is essentially just firing up all of the nodes you want with the proper [[configuration]]. One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process: +Setting up a cluster is essentially just firing up all of the nodes you want with the proper [configuration](Configuration.html). One thing to be aware of is that there are a few properties in the configuration that potentially need to be set individually for each process: ``` druid.server.type=historical|realtime From 0e3378e5cd762cb354913fce6c53b6503259bc29 Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Mon, 9 Dec 2013 15:17:56 -0800 Subject: [PATCH 3/6] refactored sidebar sections to group related topics together under a section that encompasses what they are about, and communicates to a user that this is the go-to section for provisioning/deploying etc. a druid cluster --- docs/content/toc.textile | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/content/toc.textile b/docs/content/toc.textile index 84fde0369977..3b058a95c2c3 100644 --- a/docs/content/toc.textile +++ b/docs/content/toc.textile @@ -14,15 +14,11 @@ h2. Getting Started * "Tutorial: Loading Your Data Part 2":./Tutorial:-Loading-Your-Data-Part-2.html * "Tutorial: All About Queries":./Tutorial:-All-About-Queries.html -h2. Evaluate Druid +h2. Operations * "Cluster Setup":./Cluster-setup.html -* "Booting a Production Cluster":./Booting-a-production-cluster.html - -h2. Configuration * "Configuration":Configuration.html - -h2. Extend Druid -* "Modules":./Modules.html +* "Extending Druid":./Modules.html +* "Booting a Production Cluster":./Booting-a-production-cluster.html h2. Data Ingestion * "Realtime":./Realtime.html From 19be4f8bb0217ecee986d30f59c15b38e5263281 Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Mon, 9 Dec 2013 16:08:14 -0800 Subject: [PATCH 4/6] Refactored modules section for better flow/readability and added link to extending druid --- docs/content/Configuration.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/content/Configuration.md b/docs/content/Configuration.md index 1934b941bc31..47c495f01423 100644 --- a/docs/content/Configuration.md +++ b/docs/content/Configuration.md @@ -16,7 +16,9 @@ There are three JVM parameters that we set on all of our processes: ## Modules -As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules. +As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. + +This page describes the configuration of the default modules. Node-specific configuration is discussed on each node's respective page. In addition, you can add custom modules to [extend Druid](Modules.html). Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath. From fcc8801b13214b651c5c2da665f31548b2e3ca32 Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Mon, 9 Dec 2013 16:11:06 -0800 Subject: [PATCH 5/6] added title --- docs/content/Modules.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/content/Modules.md b/docs/content/Modules.md index 17b8e5387855..b5b8a693053d 100644 --- a/docs/content/Modules.md +++ b/docs/content/Modules.md @@ -1,6 +1,9 @@ --- layout: doc_page --- + +# Extending Druid With Custom Modules + Druid version 0.6 introduces a new module system that allows for the addition of extensions at runtime. ## Specifying extensions @@ -164,4 +167,4 @@ Adding new Jersey resources to a module requires calling the following code to b ```java Jerseys.addResource(binder, NewResource.class); -``` \ No newline at end of file +``` From 60577a6e24cb0cc261acf466173a6451ef3a4a8e Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Wed, 11 Dec 2013 09:16:49 -0800 Subject: [PATCH 6/6] adjusted size of first section head to match others --- docs/content/toc.textile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/toc.textile b/docs/content/toc.textile index 3b058a95c2c3..ae0da14bcc70 100644 --- a/docs/content/toc.textile +++ b/docs/content/toc.textile @@ -3,7 +3,7 @@ -h1. Introduction +h2. Introduction * "About Druid":./ * "Concepts and Terminology":./Concepts-and-Terminology.html