Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions airflow/providers/apache/pig/example_dags/example_pig.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@
tags=['example'],
)

# [START create_pig]
run_this = PigOperator(
task_id="run_example_pig_script",
pig="ls /;",
pig_opts="-x local",
dag=dag,
)
# [END create_pig]
7 changes: 5 additions & 2 deletions airflow/providers/apache/pig/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
package-name: apache-airflow-providers-apache-pig
name: Apache Pig
description: |
`Apache Pig <https://pig.apache.org/>`__
`Apache Pig <https://pig.apache.org/>`__

versions:
- 2.0.1
Expand All @@ -33,6 +33,8 @@ additional-dependencies:
integrations:
- integration-name: Apache Pig
external-doc-url: https://pig.apache.org/
how-to-guide:
- /docs/apache-airflow-providers-apache-pig/operators.rst
logo: /integration-logos/apache/pig.png
tags: [apache]

Expand All @@ -46,7 +48,8 @@ hooks:
python-modules:
- airflow.providers.apache.pig.hooks.pig

hook-class-names: # deprecated - to be removed after providers add dependency on Airflow 2.2.0+
hook-class-names:
# deprecated - to be removed after providers add dependency on Airflow 2.2.0+
- airflow.providers.apache.pig.hooks.pig.PigCliHook

connection-types:
Expand Down
6 changes: 6 additions & 0 deletions docs/apache-airflow-providers-apache-pig/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@
Content
-------

.. toctree::
:maxdepth: 1
:caption: Guides

Operators <operators>

.. toctree::
:maxdepth: 1
:caption: References
Expand Down
32 changes: 32 additions & 0 deletions docs/apache-airflow-providers-apache-pig/operators.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.



Apache Pig Operators
====================

Apache Pig is a platform for analyzing large data sets that consists of a high-level language
for expressing data analysis programs, coupled with infrastructure for evaluating these programs.
Pig programs are amenable to substantial parallelization, which in turns enables them to handle very large data sets.

use the PigOperator to execute a pig script

.. exampleinclude:: /../../airflow/providers/apache/pig/example_dags/example_pig.py
:language: python
:start-after: [START create_pig]
:end-before: [END create_pig]