Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
427be24
used exec-maven-plugin to map maven's lifecycle phases with python's …
wikier Jun 27, 2016
23f42c3
synced version from maven
wikier Jun 27, 2016
061ef0a
synced version from maven
wikier Jun 27, 2016
98ca6f8
merged
wikier Jun 27, 2016
2acaed2
mapped package phase
wikier Jun 27, 2016
43bcc7a
fixed author metadata
wikier Jun 27, 2016
231ff54
trying to get travis build working
wikier Jun 27, 2016
1d815bb
BEAM-378: removed unnecessary packages previously introduced in ci en…
wikier Jun 28, 2016
97b52ad
reformattted file according PEP 8
wikier Jun 28, 2016
d7874fe
BEAM-378: trying to get right the version on both build and runtime
wikier Jun 28, 2016
12fedce
BEAM-378: changed completelly the strategy: version is only read on s…
wikier Jun 28, 2016
831d4b0
BEAM-378: fixed install_requires order
wikier Jun 29, 2016
d676108
BEAM-378: reverted PEP-8 change, since @silviulica pointed they curre…
wikier Jun 29, 2016
79b69fd
BEAM-378: added a comment about the default apache_beam.__version__
wikier Jun 29, 2016
1895047
BEAM-378: switched to tox as target for 'mvn test' phase
wikier Jun 29, 2016
dcb5ea3
BEAM-378: fixed tox cofig file argument
wikier Jun 29, 2016
e424fbf
BEAM-378: merged from upstream python-sdk branch
wikier Jun 29, 2016
137c4a0
BEAM-378: fixed typo
wikier Jul 1, 2016
a0a788e
BEAM-378: remove the depencency on apache_beam module from setup.py (…
wikier Jul 1, 2016
65fde8d
BEAM-378: clean-up
wikier Jul 1, 2016
c952964
BEAM-378: fixed pylint warnings
wikier Jul 1, 2016
c583094
BEAM-378: removed deprecated version usage
wikier Jul 1, 2016
b045a97
BEAM-378: brought back lxml as required to see if we get ci finally w…
wikier Jul 1, 2016
27917a0
BEAM-378: replaced by regex pom.xml parsing to get rid of lxml depend…
wikier Jul 6, 2016
456501e
BEAM-378: renamed variable
wikier Jul 6, 2016
08502f7
BEAM-378: removed non-necessary anymore return statement
wikier Jul 6, 2016
3da6884
BEAM-378: making rat happy
wikier Jul 6, 2016
3674690
BEAM-378: fixed 2-spaces indentation
wikier Jul 7, 2016
93c31df
BEAM-378: added a comment in the manifest regarding why pom.xml is in…
wikier Jul 7, 2016
72d128b
BEAM-378: handled tox installation in process-test-resources maven phase
wikier Jul 7, 2016
f0290bd
BEAM-378: fixed execution id
wikier Jul 8, 2016
ebe4d0e
BEAM-378: now that it's clear versions are not going to be aligned, r…
wikier Jul 8, 2016
aa8e9d1
BEAM-378: fixed build path using the right maven property
wikier Jul 8, 2016
49b1901
BEAM-378: kept the ',' at the end of the list according to @dhalperi …
wikier Jul 8, 2016
0a193fa
BEAM-378: trying to use virtualenv to run tox from maven everywhere
wikier Jul 8, 2016
d733f86
BEAM-378: merged from upstream
wikier Jul 8, 2016
62646b5
BEAM-378: making pylint happy again
wikier Jul 8, 2016
71b59bf
BEAM-378: remove unused import to make pylint happy again
wikier Jul 8, 2016
7f3c4cb
BEAM-378: removed trailing newlines to make pylint happy again
wikier Jul 8, 2016
36b4d80
BEAM-378: as @aaltay suggested, changed the strategy to use tox under…
wikier Jul 11, 2016
2d19489
BEAM-378: as @aaltay suggested, changed the strategy to use tox under…
wikier Jul 11, 2016
dc1e983
Merge branch 'python-sdk' of github.com:wikier/incubator-beam into py…
wikier Jul 11, 2016
a38a3a8
BEAM-378: trying to skip goal exec:java with the old maven version we…
wikier Jul 11, 2016
684cfb2
BEAM-368: another try to skip exec:java goal on jenkins
wikier Jul 11, 2016
db7dd39
BEAM-368: yet another try to get rig of exec:java goal on jenkins
wikier Jul 11, 2016
f1f354f
BEAM-368: trying to know the maven version is running at jenkins
wikier Jul 11, 2016
c7be1d3
BEAM-378: aligned tox directories with maven build
wikier Jul 12, 2016
6e616d8
BEAM-378: trying not to inherit exec:java execution in the python sdk
wikier Jul 19, 2016
69fe354
Revert "BEAM-378: trying not to inherit exec:java execution in the py…
wikier Jul 19, 2016
ec578d2
BEAM-378: another try to skip the exec:java goal in the python sdk
wikier Aug 1, 2016
6457f55
merged from upstream
wikier Aug 1, 2016
78779bb
BEAM-379: preventing warning with mkdir
wikier Aug 1, 2016
c7093e7
BEAM-378: removed unused imports
wikier Aug 1, 2016
2620bfb
BEAM-378: trying to use the execution id to skip exec:maven in a subm…
wikier Aug 1, 2016
23003ac
BEAM-378: moved exec:java execution to a non-existent phase to actual…
wikier Aug 1, 2016
a923207
BEAM-378: fied egg_base to make rat happy again
wikier Aug 1, 2016
1d9378a
BEAM-378: fixed wrong merge on setup.py
wikier Aug 2, 2016
fd7ddf0
BEAM-378: fixed wrong merge on apiclient.py
wikier Aug 2, 2016
e044d8d
BEAM-378: removed non-used import
wikier Aug 2, 2016
cb82ec1
BEAM-378: reverted to reference __init__.py
wikier Aug 2, 2016
cd9432f
BEAM-378: moved nose setup requirements from setuptools to tox, so we…
wikier Aug 2, 2016
4422207
BEAM-378: trying to fix the lateral issue on beam-runners-flink_2.10-…
wikier Aug 2, 2016
a0ecce5
Revert "BEAM-378: trying to fix the lateral issue on beam-runners-fli…
wikier Aug 3, 2016
d232892
BEAM-378: clean-up
wikier Aug 3, 2016
51b13f5
BEAM-378: upgraded dev version according the new release
wikier Aug 3, 2016
f0d79f7
BEAM-378: right license label
wikier Aug 3, 2016
be837b3
merged with upstream/python-sdk
wikier Aug 8, 2016
b404a42
BEAM-378: fixed parent version
wikier Aug 8, 2016
32a6348
Merge branch 'python-sdk' of github.com:wikier/incubator-beam into py…
wikier Aug 8, 2016
e9fc5f0
BEAM-378: fixed parent version
wikier Aug 8, 2016
403c425
BEAm-378: recovered exec:java execution id for being able to skip it …
wikier Aug 8, 2016
cf0bb30
fixed exec-maven-plugin configuration on beam-runners-flink_2.10-exam…
wikier Aug 9, 2016
4c60253
BEAM-378: applied the trick by @aaltay to skip inheritance in exec-ma…
wikier Aug 10, 2016
f7c8a13
Revert "fixed exec-maven-plugin configuration on beam-runners-flink_2…
wikier Aug 10, 2016
909889d
Revert "BEAM-378: moved nose setup requirements from setuptools to to…
wikier Aug 10, 2016
5014ed1
BEAM-378: excluded nose egg now that we are back managing its setup_r…
wikier Aug 10, 2016
82a8515
BEAM-378: actually upgraded to apache-rat-plugin 0.12
wikier Aug 10, 2016
35c6e2c
BEAM-378: more fine grained rat exclusion for nose egg
wikier Aug 10, 2016
7020e73
BEAM-378: pr clean-up
wikier Aug 10, 2016
86b3419
BEAM-378: more flexible rat exclusion for nose's egg
wikier Aug 11, 2016
bbd48b3
BEAM-378: pr clean-up
wikier Aug 11, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -778,7 +778,7 @@
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.11</version>
<version>0.12</version>
<configuration>
<reportFile>${project.build.directory}/${project.build.finalName}.rat</reportFile>
<excludeSubProjects>false</excludeSubProjects>
Expand All @@ -794,6 +794,7 @@
<exclude>**/test/resources/**/*.txt</exclude>
<exclude>**/test/**/.placeholder</exclude>
<exclude>.repository/**/*</exclude>
<exclude>**/nose-*.egg/**/*</exclude>

<!-- Default eclipse excludes neglect subprojects -->
<exclude>**/.checkstyle</exclude>
Expand Down
4 changes: 3 additions & 1 deletion sdks/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@

<modules>
<module>java</module>
<module>python</module>
</modules>

<build>
Expand All @@ -53,8 +54,9 @@
</execution>
</executions>
</plugin>

</plugins>
</pluginManagement>
</build>

</project>
</project>
19 changes: 19 additions & 0 deletions sdks/python/MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This file is used from Python to sync versions
include pom.xml
1 change: 0 additions & 1 deletion sdks/python/apache_beam/utils/dependency.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@
from apache_beam.utils import processes
from apache_beam.utils.options import GoogleCloudOptions
from apache_beam.utils.options import SetupOptions
from apache_beam.version import __version__


# Standard file names used for staging files.
Expand Down
1 change: 0 additions & 1 deletion sdks/python/apache_beam/utils/dependency_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
from apache_beam.utils.options import GoogleCloudOptions
from apache_beam.utils.options import PipelineOptions
from apache_beam.utils.options import SetupOptions
from apache_beam.version import __version__


class SetupTest(unittest.TestCase):
Expand Down
42 changes: 40 additions & 2 deletions sdks/python/apache_beam/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,44 @@
# limitations under the License.
#

"""Apache Beam SDK version information."""
"""Apache Beam SDK version information and utilities."""

__version__ = '0.3.0'

import re


__version__ = '0.3.0-incubating.dev' # TODO: PEP 440 and incubating suffix


# The following utilities are legacy code from the Maven integration;
# see BEAM-378 for further details.


# Reads the actual version from pom.xml file,
def get_version_from_pom():
with open('pom.xml', 'r') as f:
pom = f.read()
regex = (r'.*<parent>\s*'
r'<groupId>[a-z\.]+</groupId>\s*'
r'<artifactId>[a-z\-]+</artifactId>\s*'
r'<version>([0-9a-zA-Z\.\-]+)</version>.*')
pattern = re.compile(str(regex))
search = pattern.search(pom)
version = search.group(1)
version = version.replace("-SNAPSHOT", ".dev")
# TODO: PEP 440 and incubating suffix
return version


# Synchronizes apache_beam.__version__ field for later usage
def sync_version(version):
init_path = 'apache_beam/__init__.py'
regex = r'^__version__\s*=\s*".*"'
with open(init_path, "r") as f:
lines = f.readlines()
with open(init_path, "w") as f:
for line in lines:
if re.search(regex, line):
f.write(re.sub(regex, '__version__ = "%s"' % version, line))
else:
f.write(line)
169 changes: 169 additions & 0 deletions sdks/python/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python distribution will package this file and not any of its parents. Could you add a comment somewhere here that, in Python we use this to read the version string and and it is fine to package as a standalone file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I'll add a comment in the MANIFEST.in file.

<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-parent</artifactId>
<version>0.3.0-incubating-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>beam-sdks-python</artifactId>

<packaging>pom</packaging>

<name>Apache Beam :: SDKs :: Python</name>

<properties>
<python.interpreter.bin>python2</python.interpreter.bin>
<python.pip.bin>pip2</python.pip.bin>
<python.build.base>${project.build.directory}/build</python.build.base>
<python.user.base>${project.build.directory}/python</python.user.base>
</properties>

<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<!-- trick by @aaltay to skip inheritance -->
<phase>none</phase>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>setuptools-clean</id>
<phase>clean</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${python.interpreter.bin}</executable>
<arguments>
<argument>setup.py</argument>
<argument>clean</argument>
Copy link
Contributor

@dhalperi dhalperi Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how comprehensive is this clean? e.g., does it delete random files created by tox or lint, etc? I think we want to go as far as we can here.

for example, I wonder if we shouldn't be deleting all the files we added to apache rat plugin ignore below -- I don't think they're checked in.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a good point, @dhalperi

for the moment I'm just doing a python setup.py clean, but we could add any other clean tasks

</arguments>
</configuration>
</execution>
<execution>
<id>setuptools-build</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${python.interpreter.bin}</executable>
<arguments>
<argument>setup.py</argument>
<argument>build</argument>
<argument>--build-base</argument>
<argument>${python.build.base}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>setup-test-create-python-base</id>
<phase>process-test-resources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>mkdir</executable>
<arguments>
<argument>-p</argument>
<argument>${python.user.base}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>setup-test-tox</id>
<phase>process-test-resources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${python.pip.bin}</executable>
<arguments>
<argument>install</argument>
<argument>--user</argument>
<argument>--upgrade</argument>
<argument>--ignore-installed</argument>
<argument>tox</argument>
</arguments>
<environmentVariables>
<PYTHONUSERBASE>${python.user.base}</PYTHONUSERBASE>
</environmentVariables>
</configuration>
</execution>
<execution>
<id>setuptools-test</id>
<phase>test</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${python.user.base}/bin/tox</executable>
<arguments>
<argument>-e</argument>
<argument>py27</argument>
<argument>-c</argument>
<argument>tox.ini</argument>
</arguments>
<environmentVariables>
<PYTHONUSERBASE>${python.user.base}</PYTHONUSERBASE>
</environmentVariables>
</configuration>
</execution>
<execution>
<id>setuptools-sdist</id>
<phase>package</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${python.interpreter.bin}</executable>
<arguments>
<argument>setup.py</argument>
<argument>sdist</argument>
<argument>--dist-dir</argument>
<argument>${project.build.directory}</argument>
</arguments>
<environmentVariables>
<PYTHONUSERBASE>${python.user.base}</PYTHONUSERBASE>
</environmentVariables>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>
2 changes: 2 additions & 0 deletions sdks/python/setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,5 @@ verbosity=2
# fast_coders_test and typecoders_test.
exclude=fast_coders_test|typecoders_test

[egg_info]
egg_base = target
2 changes: 1 addition & 1 deletion sdks/python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,6 @@ def get_version():
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
],
license='Apache 2.0',
license='Apache License, Version 2.0',
keywords=PACKAGE_KEYWORDS,
)
3 changes: 3 additions & 0 deletions sdks/python/tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

[tox]
envlist = py27
toxworkdir={toxinidir}/target/tox
distdir={toxinidir}/target/dist
distshare={toxinidir}/target/distshare

[pep8]
# Disable all errors and warnings except for the ones related to blank lines.
Expand Down