Skip to content

[BEAM-13932][Playground] Container's user privileges #16842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pabloem merged 4 commits into from
Mar 4, 2022
Merged

[BEAM-13932][Playground] Container's user privileges #16842

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1d60f8a
[BEAM-13932][Playground]
Feb 13, 2022
bc53c0c
[BEAM-13932][Playground]
Feb 16, 2022
07b65b8
[BEAM-13932][Playground]
Feb 16, 2022
cab0a4e
Merge branch 'apache:master' into BEAM-13932-container-user
Feb 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions playground/backend/containers/go/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,16 @@ ENV HTTP_PROXY="http://127.0.0.1:8081"
ENV HTTPS_PROXY="http://127.0.0.1:8081"

COPY entrypoint.sh /
COPY proxy.sh /
# Run proxy
RUN sh /proxy.sh

# Create a user group `appgroup` and a user `appuser`
RUN groupadd --gid 20000 appgroup \
&& useradd --uid 20000 --gid appgroup --shell /bin/bash --create-home appuser
# Chown all the files to the app user.
RUN chown -R appuser:appgroup /opt/playground/backend/ && chown -R appuser:appgroup /opt/playground/prepared_folder/ \
&& chmod +x /entrypoint.sh

RUN mkdir -p /opt/playground/backend/executable_files/
Copy link
Contributor

@pavel-avilov pavel-avilov Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change the path to PREPARED_MOD_DIR env

Copy link
Contributor

@pavel-avilov pavel-avilov Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And in other cases too.

Copy link
Contributor Author

@AydarZaynutdinov AydarZaynutdinov Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PREPARED_MOD_DIR equals /opt/playground/prepared_folder/, but I need /opt/playground/backend/executable_files/

Copy link
Contributor

@pavel-avilov pavel-avilov Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sry, i meant executable_files to PIPELINES_FOLDER_NAME

Copy link
Contributor Author

@AydarZaynutdinov AydarZaynutdinov Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PIPELINES_FOLDER_NAME equals executable_files, but I need /opt/playground/backend/executable_files/


# Chown all required files to the `appuser`.
RUN chown -R appuser:appgroup /opt/playground/backend/executable_files/ \
&& chmod -R 777 /usr/local/share/ca-certificates/extra/ && chmod -R 777 /etc/ssl/certs && chmod +x /entrypoint.sh
Copy link
Contributor

@andreykus andreykus Feb 25, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this folders /usr/local/share/ca-certificates/extra/ /etc/ssl/certs needs right execute (7) ?

Copy link
Contributor Author

@AydarZaynutdinov AydarZaynutdinov Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if set 776 for the /usr/local/share/ca-certificates/extra/ and /etc/ssl/certs then I receive:

cp: cannot stat '/usr/local/share/ca-certificates/extra/mitmproxy-ca.crt': Permission denied
/usr/sbin/update-ca-certificates: 114: cd: can't cd to /etc/ssl/certs

if set 776 for the /etc/ssl/certs then I receive:

/usr/sbin/update-ca-certificates: 114: cd: can't cd to /etc/ssl/certs

if set 776 for the /usr/local/share/ca-certificates/extra/ then I receive:

cp: cannot stat '/usr/local/share/ca-certificates/extra/mitmproxy-ca.crt': Permission denied

So I guess I need to set 777 for both of them.


# Switch to appuser
USER appuser
Expand Down
4 changes: 0 additions & 4 deletions playground/backend/containers/go/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
copy {
from 'proxy.sh'
into 'build/'
}
copy {
from '../../../infrastructure/proxy/allow_list.py'
into 'build/'
Expand Down
9 changes: 9 additions & 0 deletions playground/backend/containers/go/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.

nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
do
sleep 2
done
openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
update-ca-certificates

/opt/playground/backend/server_go_backend
24 changes: 0 additions & 24 deletions playground/backend/containers/go/proxy.sh
View file
Open in desktop

This file was deleted.

13 changes: 7 additions & 6 deletions playground/backend/containers/java/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,16 +87,17 @@ ENV HTTP_PROXY="http://127.0.0.1:8081"
ENV HTTPS_PROXY="http://127.0.0.1:8081"

COPY entrypoint.sh /
COPY proxy.sh /
# Run proxy
RUN sh /proxy.sh

# Create a user group `appgroup` and a user `appuser`
RUN groupadd --gid 20000 appgroup \
&& useradd --uid 20000 --gid appgroup --shell /bin/bash --create-home appuser
# Chown all the files to the app user.
RUN chown -R appuser:appgroup /opt/playground/backend/ && chown -R appuser:appgroup /opt/apache/beam/jars/ \
&& chmod +x /entrypoint.sh

RUN mkdir -p /opt/playground/backend/executable_files/

# Chown all required files to the `appuser`.
RUN chown -R appuser:appgroup /opt/playground/backend/executable_files/ && chmod -R 775 /opt/apache/beam/jars/ \
&& chmod -R 777 /usr/local/share/ca-certificates/extra/ && chmod -R 777 /usr/local/openjdk-8/jre/lib/security/ \
&& chmod -R 777 /etc/ssl/certs && chmod +x /entrypoint.sh

# Switch to appuser
USER appuser
Expand Down
4 changes: 0 additions & 4 deletions playground/backend/containers/java/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
copy {
from 'proxy.sh'
into 'build/'
}
copy {
from '../../../infrastructure/proxy/allow_list.py'
into 'build/'
Expand Down
9 changes: 9 additions & 0 deletions playground/backend/containers/java/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.

nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
do
sleep 2
done
openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
update-ca-certificates

/opt/playground/backend/server_java_backend
24 changes: 0 additions & 24 deletions playground/backend/containers/java/proxy.sh
View file
Open in desktop

This file was deleted.

12 changes: 7 additions & 5 deletions playground/backend/containers/python/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,17 @@ ENV HTTP_PROXY="http://127.0.0.1:8081"
ENV HTTPS_PROXY="http://127.0.0.1:8081"

COPY entrypoint.sh /
COPY proxy.sh /
# Run proxy
RUN sh /proxy.sh

# Create a user group `appgroup` and a user `appuser`
RUN groupadd --gid 20000 appgroup \
&& useradd --uid 20000 --gid appgroup --shell /bin/bash --create-home appuser
# Chown all the files to the app user.
RUN chown -R appuser:appgroup /opt/playground/backend/ && chmod +x /entrypoint.sh

RUN mkdir -p /opt/playground/backend/executable_files/

# Chown all required files to the `appuser`.
RUN chown -R appuser:appgroup /opt/playground/backend/executable_files/ \
&& chmod 777 /usr/local/lib/python3.7/site-packages/certifi/cacert.pem \
&& chmod -R 777 /usr/local/share/ca-certificates/extra/ && chmod -R 777 /etc/ssl/certs && chmod +x /entrypoint.sh

# Switch to appuser
USER appuser
Expand Down
4 changes: 0 additions & 4 deletions playground/backend/containers/python/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
copy {
from 'proxy.sh'
into 'build/'
}
copy {
from '../../../infrastructure/proxy/allow_list.py'
into 'build/'
Expand Down
10 changes: 10 additions & 0 deletions playground/backend/containers/python/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.

nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
do
sleep 2
done
openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
update-ca-certificates
cat /home/appuser/.mitmproxy/mitmproxy-ca.pem >> /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
Copy link
Contributor

@andreykus andreykus Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needed only on python env. ?

Copy link
Contributor Author

@AydarZaynutdinov AydarZaynutdinov Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this part:

openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
update-ca-certificates

is a common for all SDK that uses a proxy, but this one:

cat /home/appuser/.mitmproxy/mitmproxy-ca.pem >> /usr/local/lib/python3.7/site-packages/certifi/cacert.pem

is used only for a Python SDK.

Copy link
Contributor

@andreykus andreykus Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


/opt/playground/backend/server_python_backend
25 changes: 0 additions & 25 deletions playground/backend/containers/python/proxy.sh
View file
Open in desktop

This file was deleted.

12 changes: 7 additions & 5 deletions playground/backend/containers/scio/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,17 @@ ENV HTTP_PROXY="http://127.0.0.1:8081"
ENV HTTPS_PROXY="http://127.0.0.1:8081"

COPY entrypoint.sh /
COPY proxy.sh /
# Run proxy
RUN sh /proxy.sh

# Create a user group `appgroup` and a user `appuser`
RUN groupadd --gid 20000 appgroup \
&& useradd --uid 20000 --gid appgroup --shell /bin/bash --create-home appuser
# Chown all the files to the app user.
RUN chown -R appuser:appgroup /opt/playground/backend/ && chmod +x /entrypoint.sh

RUN mkdir -p /opt/playground/backend/executable_files/

# Chown all required files to the `appuser`.
RUN chown -R appuser:appgroup /opt/playground/backend/executable_files/ \
&& chmod -R 777 /usr/local/share/ca-certificates/extra/ && chmod -R 777 /usr/local/openjdk-8/jre/lib/security/ \
&& chmod -R 777 /etc/ssl/certs && chmod +x /entrypoint.sh

# Switch to appuser
USER appuser
Expand Down
4 changes: 0 additions & 4 deletions playground/backend/containers/scio/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
copy {
from 'proxy.sh'
into 'build/'
}
copy {
from '../../../infrastructure/proxy/allow_list.py'
into 'build/'
Expand Down
9 changes: 9 additions & 0 deletions playground/backend/containers/scio/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.

nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
do
sleep 2
done
openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
update-ca-certificates

/opt/playground/backend/server_scio_backend
24 changes: 0 additions & 24 deletions playground/backend/containers/scio/proxy.sh
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions playground/infrastructure/proxy/allow_list.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@
"dataflow-samples",
"beam-samples",
"apache-beam-samples",
"playground-precompiled-objects",
]
6 changes: 4 additions & 2 deletions playground/infrastructure/proxy/allow_list_proxy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@

def request(flow: http.HTTPFlow) -> None:
allowed_bucket = flow.request.pretty_host == GCS_HOST and \
flow.request.path.split("/")[1] in ALLOWED_BUCKET_LIST
(flow.request.path.split("/")[1] in ALLOWED_BUCKET_LIST or \
flow.request.path.split("/")[4] in ALLOWED_BUCKET_LIST)
allowed_host = flow.request.pretty_host in ALLOWED_LIST
if not (allowed_bucket or allowed_host):
flow.response = http.Response.make(
status_code=403,
content="Making requests to the hosts that are not listed "
"in the allowed list is forbidden.")
"in the allowed list is forbidden. "
"host:" + flow.request.pretty_host + ", path: " + flow.request.path)