Skip to content

Removed Embedded python install #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ader1990 merged 1 commit into from
Apr 14, 2020
Merged

Removed Embedded python install #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 10 additions & 16 deletions scripts/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
### Requirements ###

#### Clean Windows Server 2016 /2019 or Windows 10 install with latest updates
#### Clean Windows Server 2016 / 2019 or Windows 10 install with latest updates

#### Python installed and added to path (with pip installed) if PythonOrigin is set to AlreadyInstalled.
Download link: https://www.python.org/ftp/python/3.7.7/python-3.7.7-amd64.exe
Expand All @@ -9,24 +9,28 @@
- Programming Languages -> Visual C++ (all)
- Windows and Web Development -> Windows 8.1 (only Tools and Windows SDKs)

VS 2015 Download link (you need to be logged in to access it): https://download.my.visualstudio.com/db/en_visual_studio_community_2015_with_update_1_x86_x64_web_installer_8234321.exe
VS 2015 Download page (you need to be logged in to access it): https://visualstudio.microsoft.com/vs/older-downloads/
If you prefer to not build pywin32, Visual Studio 2017 / 2019 can be used too.
Reboot after VS installation is complete.

#### To build Python from source, Visual Studio 2017 Community installed with the following components:
- VC++ 2017 v141
- Windows 10 SDK (10.0.14393.0). This version supports Python 3.8 build too.
- Windows 10 SDK (10.0.14393.0). This version of VS supports Python 3.8 build too.

#### All the Python packages will be built and installed using pip flags: "--no-binary :all:"

#### The full build of cloudbase-init (and dependencies) using Embedded Python takes around 10 minutes.

#### The full build of cloudbase-init (and dependencies) using Python from source takes around 20 minutes.

#### How to run:


```powershell
# install Cloudbase-Init using Python already installed (Python and Python scripts folders should be added to path).
.\build_install_from_sources.ps1

# install Cloudbase-Init using Python from source (tag v3.7.7)
.\build_install_from_sources.ps1 -PythonOrigin FromSource -PythonVersion "v3.7.7"

# full command line
.\build_install_from_sources.ps1 `
-CloudbaseInitRepoUrl "https://github.com/cloudbase/cloudbase-init" `
Expand All @@ -39,15 +43,6 @@
-WheelSourceUrl "https://github.com/pypa/wheel/archive/0.34.2.tar.gz" `,
-CleanBuildArtifacts:$false,
-BuildDir "build"

# install Cloudbase-Init using Python already installed (Python and Python scripts folders should be added to path).
.\build_install_from_sources.ps1

# install Cloudbase-Init using Python from source (tag v3.7.7)
.\build_install_from_sources.ps1 -PythonOrigin FromSource -PythonVersion "v3.7.7"

# install Cloudbase-Init using Python Embedded 3.7.7
.\build_install_from_sources.ps1 -PythonOrigin Embedded -PythonVersion "3.7.7"
```


Expand All @@ -56,8 +51,7 @@
- Create / clean temporary build directory
- If PythonOrigin="AlreadyInstalled" is set, do nothing. Python should be already installed and added to path.
- If PythonOrigin="FromSource" is set, download the Python source from GitHub, build, prepare it and add it to path. Setuptools, pip and wheel will be built from source and installed.
- If PythonOrigin="Embedded" is set, download the Python embedded, prepare it and add it to path. Setuptools, pip and wheel will be built from source and installed.
- Build and install PyWin32 from sources
- Build and install PyMI from sources
- Build, install and create Cloudbase-Init binary from sources
- If CleanBuildArtifacts is set and PythonOrigin is Embedded or FromSource, cleanup the .pdb, .pyc, header files.
- If CleanBuildArtifacts is set and PythonOrigin is FromSource, cleanup the .pdb, .pyc, header files.
111 changes: 9 additions & 102 deletions scripts/build_install_from_sources.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,18 @@ param(
[string]$CloudbaseInitRepoUrl="https://github.com/cloudbase/cloudbase-init",
[string]$PyWin32RepoUrl="https://github.com/mhammond/pywin32",
[string]$PyMiRepoUrl="https://github.com/cloudbase/PyMI",
[ValidateSet("AlreadyInstalled", "Embedded", "FromSource")]
[ValidateSet("AlreadyInstalled", "FromSource")]
[string]$PythonOrigin="AlreadyInstalled",
# Specifies which version to use in case Embedded or FromSource is used
# In Embedded case: "https://www.python.org/ftp/python/$PythonVersion" should exist
# Specifies which version to use in case FromSource is used
# In FromSource case, the GitHub https://github.com/python/cpython branch should exist (tags also work as branches)
[string]$PythonVersion="v3.7.7",
# Specifies which setuptools git source to use in case Embedded or FromSource is used
# Specifies which setuptools git source to use in case FromSource is used
[string]$SetuptoolsGitUrl="https://github.com/pypa/setuptools",
# Specifies which pip source in ttar.gz format to use in case Embedded or FromSource is used
# Specifies which pip source in tar.gz format to use in case FromSource is used
[string]$PipSourceUrl="https://github.com/pypa/pip/archive/20.0.2.tar.gz",
# Specifies which wheel source in ttar.gz format to use in case Embedded or FromSource is used
# Specifies which wheel source in tar.gz format to use in case FromSource is used
[string]$WheelSourceUrl="https://github.com/pypa/wheel/archive/0.34.2.tar.gz",
# If the PythonOrigin is set to Embedded or FromSource, clean the Python folder (remove .pdb, .pyc, header files)
# If the PythonOrigin is set to FromSource, clean the Python folder (remove .pdb, .pyc, header files)
[switch]$CleanBuildArtifacts,
[string]$BuildDir=""
)
Expand Down Expand Up @@ -193,23 +192,6 @@ function Expand-Archive {
}
}

function Setup-PythonPackage {
param([string]$SourcePath)

Write-Host "Setup Python package from ${SourcePath}"

$cmdArgs = @("-W ignore", "setup.py", "install")

Run-CmdWithRetry {
try {
Push-Location (Join-Path $BuildDir $SourcePath)
Run-Command -Cmd "python" -Arguments $cmdArgs
} finally {
Pop-Location
}
}
}

function Prepare-BuildDir {
Write-Host "Creating / Cleaning up build directory ${BuildDir}"

Expand Down Expand Up @@ -239,15 +221,6 @@ function Install-PyWin32FromSource {
Install-PythonPackage -SourcePath $sourcePath
}

function Install-ComtypesFromSource {
param($Url)

$sourcePath = "comptype"

Clone-Repo $Url $sourcePath
Install-PythonPackage -SourcePath $sourcePath
}

function Install-PyMI {
param($Url)

Expand Down Expand Up @@ -291,68 +264,6 @@ function Install-SetuptoolsFromSource {
}
}

function Setup-EmbeddedPythonEnvironment {
param($EmbeddedPythonVersion)

$EmbeddedPythonUrl = "https://www.python.org/ftp/python/${EmbeddedPythonVersion}/python-${EmbeddedPythonVersion}-embed-amd64.zip"
$SourcePythonUrl = "https://www.python.org/ftp/python/${EmbeddedPythonVersion}/Python-${EmbeddedPythonVersion}.tgz"
$pythonVersionHeader = "python" + (($EmbeddedPythonVersion.split(".") | Select-Object -First 2) -Join "")

Download-File $EmbeddedPythonUrl "${PythonDir}.zip"
New-Item -Type Directory -Path $PythonDir
Expand-Archive "${PythonDir}.zip" $PythonDir
Remove-Item -Force "${PythonDir}.zip"

$sourcePythonDir = "$BuildDir\source-python"
Download-File "${SourcePythonUrl}" "${sourcePythonDir}.tgz"
New-Item -Type Directory -Path $sourcePythonDir
Expand-Archive "${sourcePythonDir}.tgz" $sourcePythonDir
Remove-Item -Force "${sourcePythonDir}.tgz"
New-Item -Type Directory -Path "$sourcePythonDir\src"
Expand-Archive "${sourcePythonDir}\source-python.tar" "$sourcePythonDir\src"
Remove-Item -Force -Recurse "${sourcePythonDir}\source-python.tar"
$sourcePythonDir = "${sourcePythonDir}\src\Python-${EmbeddedPythonVersion}"

Remove-Item -Force "${PythonDir}\${pythonVersionHeader}._pth"

New-Item -Type Directory -Path "${PythonDir}\Lib"
Expand-Archive "${PythonDir}\${pythonVersionHeader}.zip" "${PythonDir}\Lib"
Remove-Item -Force "${PythonDir}\${pythonVersionHeader}.zip"

Copy-Item -Recurse -Force "${sourcePythonDir}\Include" "${PythonDir}\"
Copy-Item -Recurse -Force "${sourcePythonDir}\PC\pyconfig.h" "${PythonDir}\Include\"

New-Item -Type Directory -Path "${PythonDir}\libs\"
$pythonLibPath = "${PythonDir}\libs\${pythonVersionHeader}.lib"
$pythonLibDefRawPath = "${PythonDir}\libs\${pythonVersionHeader}.raw.def"
$pythonLibDefPath = "${PythonDir}\libs\${pythonVersionHeader}.def"
$pythonDllPath = "${PythonDir}\${pythonVersionHeader}.dll"
dumpbin.exe /exports "${pythonDllPath}" > "${pythonLibDefRawPath}"
if ($LastExitCode) {
throw "Failed to dump symbols"
}
Write-Output "EXPORTS" > "${pythonLibDefPath}"
Get-Content "${pythonLibDefRawPath}" | ForEach-Object {
$splitObject = $_.split(" ")
[array]::reverse($splitObject)
if ($splitObject[0] -clike "*Py*") {
Write-Output $splitObject[0] >> "${pythonLibDefPath}"
}
}
lib.exe /def:"${pythonLibDefPath}" /out:"${pythonLibPath}" /machine:x64
if ($LastExitCode) {
throw "Failed to create symbol map"
}
Remove-Item -Force $pythonLibDefRawPath
Remove-Item -Force $pythonLibDefPath

# Embedded Python has bdist_wininst.pyc reimplemented to throw an error if any package using this feature
# is to be installed. Pywin32 and comtypes packages cannot be installed with pip or by running setup.py install.
$bdistFile = "Lib\distutils\command\bdist_wininst.py"
Copy-Item -Force "${sourcePythonDir}\${bdistFile}" "${PythonDir}\${bdistFile}"
Remove-Item -Force "${PythonDir}\${bdistFile}c"
}

function Setup-FromSourcePythonEnvironment {
param($FromSourcePythonVersion)

Expand Down Expand Up @@ -426,15 +337,11 @@ try {
# Setup pip upper requirements
Setup-PythonPip

if (@("Embedded", "FromSource") -contains $PythonOrigin) {
if (@("FromSource") -contains $PythonOrigin) {
if (!($PythonVersion -and $SetuptoolsGitUrl -and $PipSourceUrl -and $WheelSourceUrl)) {
throw "If PythonOrigin is set to ${PythonOrigin}, SetuptoolsGitUrl, PipSourceUrl and WheelSourceUrl must be set"
}
if ($PythonOrigin -eq "Embedded") {
Setup-EmbeddedPythonEnvironment $PythonVersion
} else {
Setup-FromSourcePythonEnvironment $PythonVersion
}
Setup-FromSourcePythonEnvironment $PythonVersion
$env:path = "${PythonDir};${PythonDir}\scripts;" + $env:path
}

Expand All @@ -450,7 +357,7 @@ try {

Install-CloudbaseInit $CloudbaseInitRepoUrl

if ($CleanBuildArtifacts -and (@("Embedded", "FromSource") -contains $PythonOrigin)) {
if ($CleanBuildArtifacts -and (@("FromSource") -contains $PythonOrigin)) {
Clean-BuildArtifacts
}
} finally {
Expand Down