tez.lib.uris /hdp/apps/${hdp.version}/tez/tez.tar.gz Comma-delimited list of the location of the Tez libraries which will be localized for DAGs. Specifying a single .tar.gz or .tgz assumes that a compressed version of the tez libs is being used. This is uncompressed into a tezlibs directory when running containers, and tezlibs/;tezlibs/lib/ are added to the classpath (after . and .*). If multiple files are specified - files are localized as regular files, contents of directories are localized as regular files (non-recursive). tez.cluster.additional.classpath.prefix /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure tez.am.log.level INFO Root Logging level passed to the Tez app master tez.generate.debug.artifacts false Generate debug artifacts such as a text representation of the submitted DAG plan tez.staging-dir /tmp/${user.name}/staging The staging dir used while submitting DAGs tez.am.resource.memory.mb TODO-CALCULATE-MEMORY-SETTINGS The amount of memory to be used by the AppMaster. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version} Cluster default Java options for the Tez AppMaster process. These will be prepended to the properties specified via tez.am.launch.cmd-opts tez.am.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC Java options for the Tez AppMaster process. The Xmx value is derived based on tez.am.resource.memory.mb and is 80% of the value by default. Used only if the value is not specified explicitly by the DAG definition. tez.am.launch.env LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64 Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.task.resource.memory.mb 1536 The amount of memory to be used by launched tasks. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.cluster-default.cmd-opts -server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version} Cluster default Java options for tasks. These will be prepended to the properties specified via tez.task.launch.cmd-opts tez.task.launch.cmd-opts -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC Java options for tasks. The Xmx value is derived based on tez.task.resource.memory.mb and is 80% of this value by default. Used only if the value is not specified explicitly by the DAG definition. tez.task.launch.env LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64 Additional execution environment entries for tez. This is not an additive property. You must preserve the original value if you want to have access to native libraries. Used only if the value is not specified explicitly by the DAG definition. tez.shuffle-vertex-manager.min-src-fraction 0.2 In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are schedule tez.shuffle-vertex-manager.max-src-fraction 0.4 In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction tez.am.am-rm.heartbeat.interval-ms.max 250 The maximum heartbeat interval between the AM and RM in milliseconds tez.grouping.split-waves 1.7 The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue tez.grouping.min-size 16777216 Lower bound on the size (in bytes) of a grouped split, to avoid generating too many splits tez.grouping.max-size 1073741824 Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large split tez.am.container.reuse.enabled true Configuration to specify whether container should be reused tez.am.container.reuse.rack-fallback.enabled true Whether to reuse containers for rack local tasks. Active only if reuse is enabled tez.am.container.reuse.non-local-fallback.enabled false Whether to reuse containers for non-local tasks. Active only if reuse is enabled tez.am.container.idle.release-timeout-min.millis 10000 The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled. tez.am.container.idle.release-timeout-max.millis 20000 The maximum amount of time to hold on to a container if no task can be assigned to it immediately. Only active when reuse is enabled. tez.am.container.reuse.locality.delay-allocation-millis 250 The amount of time to wait before assigning a container to the next level of locality. NODE -> RACK -> NON_LOCAL tez.am.max.app.attempts 2 Specifies the total number of time the app master will run in case recovery is triggered tez.am.maxtaskfailures.per.node 10 The maximum number of allowed task attempt failures on a node before it gets marked as blacklisted tez.task.am.heartbeat.counter.interval-ms.max 4000 Time interval at which task counters are sent to the AM tez.task.get-task.sleep.interval-ms.max 200 The maximum amount of time, in seconds, to wait before a task asks an AM for another task tez.task.max-events-per-heartbeat 500 Maximum number of of events to fetch from the AM by the tasks in a single heartbeat. tez.session.client.timeout.secs -1 Time (in seconds) to wait for AM to come up when trying to submit a DAG from the client tez.session.am.dag.submit.timeout.secs 300 Time (in seconds) for which the Tez AM should wait for a DAG to be submitted before shutting down tez.counters.max 2000 The number of allowed counters for the executing DAG tez.counters.max.groups 1000 The number of allowed counter groups for the executing DAG tez.runtime.compress true Whether intermediate data should be compressed or not tez.runtime.compress.codec org.apache.hadoop.io.compress.SnappyCodec The coded to be used if compressing intermediate data. Only applicable if tez.runtime.compress is enabled tez.runtime.io.sort.mb 512 The size of the sort buffer when output needs to be sorted tez.runtime.unordered.output.buffer.size-mb 100 The size of the buffer when output does not require to be sorted