Add dev docker compose #50

turbaszek · 2020-10-10T21:12:17Z

I would love to make this to work out of the box but this will require #48. Otherwise, the setup will be so confusing...

Depends on #48
Related to #40

setup/setup.py

turbaszek · 2020-10-11T21:17:52Z

Once we have #48 and this PR anyone will be able to start Apache Kibble by simply running:

docker-compose -f docker-compose-dev.yaml up setup
docker-compose -f docker-compose-dev.yaml up ui

This should be enough for development purposes 👍

michalslowikowski00 · 2020-10-12T06:40:02Z

Great. I like it.

docker-compose -f docker-compose-dev.yaml up setup
docker-compose -f docker-compose-dev.yaml up ui

turbaszek · 2020-10-12T17:15:04Z

docker-compose-dev.yaml

+
+  # Apache Kibble web ui server
+  ui:
+    image: nginx:latest


I decided to use nginx as I'm more familiar with it than with httpd

turbaszek · 2020-10-12T17:15:53Z

@sharanf @Humbedooh happy to get a review from you 🎉

turbaszek · 2020-10-14T12:33:16Z

Hi all, is there anything that should be done to get this PR merged? @Humbedooh @sharanf

sharanf · 2020-10-16T09:20:43Z

@turbaszek I see #48 is already merged and it is showing a conflct here. Please can you take a look.

turbaszek · 2020-10-16T10:27:05Z

Will need #58 to make CI green

turbaszek · 2020-10-16T10:45:52Z

@Humbedooh @sharanf this one is rebased and ready for your review

sharanf · 2020-10-17T14:18:47Z

@turbaszek - still looking at this as I haven't been able to get it to run locally on my machine yet.

turbaszek · 2020-10-17T15:59:17Z

@turbaszek - still looking at this as I haven't been able to get it to run locally on my machine yet.

No problem. You may find this tool useful:
https://github.com/cli/cli

Once you have github cli tool it's enough to do in your local Apache Kibble repository:

gh pr checkout 50

to checkout this PR and test everything locally 👍

sharanf · 2020-10-19T17:38:00Z

@turbaszek - still looking at this as I haven't been able to get it to run locally on my machine yet.

No problem. You may find this tool useful:
https://github.com/cli/cli

Once you have github cli tool it's enough to do in your local Apache Kibble repository:
gh pr checkout 50
to checkout this PR and test everything locally +1

Thanks @turbaszek, I installed the github cli tool and am still getting an error. This time it is "ERROR: pull access denied for kibble, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" so it could be something I have missed as part of the setup. I will have another try later in the week but please don't let me hold things up for you if you are happy things are working OK.

turbaszek · 2020-10-19T18:34:24Z

This time it is "ERROR: pull access denied for kibble, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" so it could be something I have missed as part of the setup.

@sharanf good catch! I will fix it as this is a real problem

sharanf · 2020-10-20T15:39:29Z

This time it is "ERROR: pull access denied for kibble, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" so it could be something I have missed as part of the setup.

@sharanf good catch! I will fix it as this is a real problem

Hi @turbaszek I have had another go and it got a lot further this time.
It exited with the following error " Index creation failed: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f61ebb232b0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f61ebb232b0>: Failed to establish a new connection: [Errno -2] Name or service not known)
kibble_setup_1 exited with code 1"
Once again not sure if this is a real problem or my setup! :-)

Humbedooh · 2020-10-20T15:47:36Z

This time it is "ERROR: pull access denied for kibble, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" so it could be something I have missed as part of the setup.

@sharanf good catch! I will fix it as this is a real problem

Hi @turbaszek I have had another go and it got a lot further this time.
It exited with the following error " Index creation failed: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f61ebb232b0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f61ebb232b0>: Failed to establish a new connection: [Errno -2] Name or service not known)
kibble_setup_1 exited with code 1"
Once again not sure if this is a real problem or my setup! :-)

Sounds like ElasticSearch isn't running...

turbaszek · 2020-10-20T17:11:23Z

docker-compose-dev.yaml

+    build:
+      context: .
+      dockerfile: Dockerfile.dev
+    command: bash -c "sleep 25 && python setup/setup.py -e elasticsearch -a -k"


Sounds like ElasticSearch isn't running...

That's why there is this sleep... it's a hack but usually worked. I can figure a better health check (this is tricky in docker compose) or we can increase this value for time being

Or I can add some loop in python code to retry it few times

turbaszek · 2020-10-20T19:09:38Z

Should be good now:

➜ docker-compose -f docker-compose-dev.yaml up setup
Creating network "kibble_default" with the default driver
Creating volume "kibble_kibble-es-data" with default driver
Creating kibble_elasticsearch_1 ... done
Creating kibble_setup_1         ... done
Attaching to kibble_setup_1
setup_1          | Welcome to the Apache Kibble setup script!
setup_1          | Configuring Apache Kibble elasticsearch instance with the following arguments:
setup_1          | - hostname: elasticsearch
setup_1          | - port: 9200
setup_1          | - dbname: kibble
setup_1          | - shards: 5
setup_1          | - replicas: 1
setup_1          |
setup_1          | Trying to create ES index...
setup_1          | Trying to create ES index...
setup_1          | Trying to create ES index...
setup_1          | Creating index kibble_api
setup_1          | Creating index kibble_ci_build
setup_1          | Creating index kibble_ci_queue
setup_1          | Creating index kibble_code_commit
setup_1          | Creating index kibble_code_commit_unique
setup_1          | Creating index kibble_code_modification
setup_1          | Creating index kibble_evolution
setup_1          | Creating index kibble_file_history
setup_1          | Creating index kibble_forum_post
setup_1          | Creating index kibble_forum_topic
setup_1          | Creating index kibble_ghstats
setup_1          | Creating index kibble_im_stats
setup_1          | Creating index kibble_im_ops
setup_1          | Creating index kibble_im_msg
setup_1          | Creating index kibble_issue
setup_1          | Creating index kibble_logstats
setup_1          | Creating index kibble_email
setup_1          | Creating index kibble_mailstats
setup_1          | Creating index kibble_mailtop
setup_1          | Creating index kibble_organisation
setup_1          | Creating index kibble_view
setup_1          | Creating index kibble_publish
setup_1          | Creating index kibble_source
setup_1          | Creating index kibble_stats
setup_1          | Creating index kibble_social_follow
setup_1          | Creating index kibble_social_followers
setup_1          | Creating index kibble_social_follower
setup_1          | Creating index kibble_social_person
setup_1          | Creating index kibble_uisession
setup_1          | Creating index kibble_useraccount
setup_1          | Creating index kibble_message
setup_1          | Creating index kibble_person
setup_1          | Indices created!
setup_1          |
setup_1          | Creating administrator account
setup_1          | Account created!
setup_1          |
setup_1          | /kibble/setup/../api/yaml/kibble.yaml already exists! Writing to /kibble/setup/../api/yaml/kibble.yaml.tmp instead
setup_1          | Writing Kibble config to /kibble/setup/../api/yaml/kibble.yaml.tmp
setup_1          |
setup_1          | All done, Kibble should...work now :)
kibble_setup_1 exited with code 0

@sharanf you may need to do docker-compose -f docker-compose-dev.yaml build setup before retrying

sharanf · 2020-10-20T19:42:08Z

@turbaszek I did the the build setup and got a whole load of text followed by:

Successfully built 46142dc0a961
Successfully tagged apache/kibble:latest

Then ran the docker-compose -f docker-compose-dev.yaml up setup
Starting kibble_elasticsearch_1 ...
Starting kibble_elasticsearch_1 ... done
Recreating kibble_setup_1 ...
Recreating kibble_setup_1 ... done
Attaching to kibble_setup_1
setup_1 | Welcome to the Apache Kibble setup script!
setup_1 | Configuring Apache Kibble elasticsearch instance with the following arguments:
setup_1 | - hostname: elasticsearch
setup_1 | - port: 9200
setup_1 | - dbname: kibble
setup_1 | - shards: 5
setup_1 | - replicas: 1
setup_1 |
setup_1 | Trying to create ES index...
setup_1 | Trying to create ES index...
setup_1 | Trying to create ES index...
setup_1 | Trying to create ES index...
setup_1 | Trying to create ES index...
setup_1 | Traceback (most recent call last):
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
setup_1 | (self._dns_host, self.port), self.timeout, **extra_kw
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
setup_1 | for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
setup_1 | File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
setup_1 | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
setup_1 | socket.gaierror: [Errno -2] Name or service not known
setup_1 |
setup_1 | During handling of the above exception, another exception occurred:
setup_1 |
setup_1 | Traceback (most recent call last):
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 246, in perform_request
setup_1 | method, url, body, retries=Retry(False), headers=request_headers, **kw
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
setup_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
setup_1 | raise six.reraise(type(error), error, _stacktrace)
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
setup_1 | raise value
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
setup_1 | chunked=chunked,
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
setup_1 | conn.request(method, url, **httplib_request_kw)
setup_1 | File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
setup_1 | self._send_request(method, url, body, headers, encode_chunked)
setup_1 | File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
setup_1 | self.endheaders(body, encode_chunked=encode_chunked)
setup_1 | File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
setup_1 | self._send_output(message_body, encode_chunked=encode_chunked)
setup_1 | File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
setup_1 | self.send(msg)
setup_1 | File "/usr/local/lib/python3.6/http/client.py", line 980, in send
setup_1 | self.connect()
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
setup_1 | conn = self._new_conn()
setup_1 | File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
setup_1 | self, "Failed to establish a new connection: %s" % e
setup_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f74222474e0>: Failed to establish a new connection: [Errno -2] Name or service not known
setup_1 |
setup_1 | During handling of the above exception, another exception occurred:
setup_1 |
setup_1 | Traceback (most recent call last):
setup_1 | File "setup/setup.py", line 346, in
setup_1 | main()
setup_1 | File "setup/setup.py", line 318, in main
setup_1 | reraise=True
setup_1 | File "/usr/local/lib/python3.6/site-packages/tenacity/init.py", line 390, in iter
setup_1 | do = self.iter(retry_state=retry_state)
setup_1 | File "/usr/local/lib/python3.6/site-packages/tenacity/init.py", line 368, in iter
setup_1 | raise retry_exc.reraise()
setup_1 | File "/usr/local/lib/python3.6/site-packages/tenacity/init.py", line 186, in reraise
setup_1 | raise self.last_attempt.result()
setup_1 | File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
setup_1 | return self.__get_result()
setup_1 | File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
setup_1 | raise self._exception
setup_1 | File "setup/setup.py", line 330, in main
setup_1 | skiponexist=args.skiponexist,
setup_1 | File "setup/setup.py", line 107, in create_es_index
setup_1 | es_version =es.info()['version']['number']
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
setup_1 | return func(*args, params=params, headers=headers, **kwargs)
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/init.py", line 285, in info
setup_1 | "GET", "/", params=params, headers=headers
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 390, in perform_request
setup_1 | raise e
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 365, in perform_request
setup_1 | timeout=timeout,
setup_1 | File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 258, in perform_request
setup_1 | raise ConnectionError("N/A", str(e), e)
setup_1 | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f74222474e0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f74222474e0>: Failed to establish a new connection: [Errno -2] Name or service not known)
kibble_setup_1 exited with code 1

turbaszek · 2020-10-20T19:48:42Z

@sharanf can I ask you to do two things:

Add this print(f"Elasticsearch: {args.hostname}:{args.port}") right after line 313 in setup/setup.py - this will help us verify that it works as expected
Can you show us your kibble config? cat api/yaml/kibble.yaml

Apart from that can you do:

docker-compose -f docker-compose-dev.yaml up elasticsearch

This should show us if elasticsearch is working 👍

turbaszek · 2020-10-23T12:18:52Z

As discussed on #kibble slack we are good with merging this one and fixing potential problems in follow up PRs 👌

fixup! Add development docker-compose fixup! fixup! Add development docker-compose fixup! fixup! fixup! Add development docker-compose

turbaszek force-pushed the add-dev-docker-compose branch 3 times, most recently from 908fd1d to 55b7caf Compare October 10, 2020 22:29

turbaszek mentioned this pull request Oct 10, 2020

Make api a package #51

Merged

turbaszek force-pushed the add-dev-docker-compose branch 2 times, most recently from 156da6f to cc283c3 Compare October 11, 2020 08:34

turbaszek mentioned this pull request Oct 11, 2020

Refactor setup.py #48

Merged

michalslowikowski00 reviewed Oct 11, 2020

View reviewed changes

setup/setup.py Show resolved Hide resolved

turbaszek force-pushed the add-dev-docker-compose branch from 00e7e91 to 1902ce3 Compare October 12, 2020 12:20

turbaszek marked this pull request as ready for review October 12, 2020 12:31

turbaszek commented Oct 12, 2020

View reviewed changes

turbaszek force-pushed the add-dev-docker-compose branch from 1902ce3 to 1523b11 Compare October 16, 2020 09:54

turbaszek force-pushed the add-dev-docker-compose branch from 1523b11 to 77333f4 Compare October 16, 2020 10:44

sharanf self-requested a review October 16, 2020 13:18

turbaszek added the area:dev label Oct 18, 2020

turbaszek requested a review from Humbedooh October 18, 2020 11:33

turbaszek commented Oct 20, 2020

View reviewed changes

turbaszek added 6 commits October 23, 2020 14:19

Add development docker-compose

d605094

fixup! Add development docker-compose fixup! fixup! Add development docker-compose fixup! fixup! fixup! Add development docker-compose

fixup! Add development docker-compose

55df9fb

fixup! fixup! Add development docker-compose

ded5be8

fixup! fixup! fixup! Add development docker-compose

5b80b64

fixup! fixup! fixup! fixup! Add development docker-compose

52057e6

Add license header

a57a680

turbaszek force-pushed the add-dev-docker-compose branch from 3b37a97 to a57a680 Compare October 23, 2020 12:20

turbaszek merged commit 7092e86 into apache:master Oct 23, 2020

turbaszek deleted the add-dev-docker-compose branch October 23, 2020 12:22

sharanf mentioned this pull request Oct 25, 2020

Problems installing Kibble using Docker compose #79

Open

turbaszek mentioned this pull request Nov 13, 2020

Refactor setup scripts as kibble cli command #91

Merged

Add dev docker compose #50

Add dev docker compose #50

Uh oh!

Conversation

turbaszek commented Oct 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

turbaszek commented Oct 11, 2020

Uh oh!

michalslowikowski00 commented Oct 12, 2020

Uh oh!

turbaszek Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

turbaszek commented Oct 12, 2020

Uh oh!

turbaszek commented Oct 14, 2020

Uh oh!

sharanf commented Oct 16, 2020

Uh oh!

turbaszek commented Oct 16, 2020

Uh oh!

turbaszek commented Oct 16, 2020

Uh oh!

sharanf commented Oct 17, 2020

Uh oh!

turbaszek commented Oct 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharanf commented Oct 19, 2020

Uh oh!

turbaszek commented Oct 19, 2020

Uh oh!

sharanf commented Oct 20, 2020

Uh oh!

Humbedooh commented Oct 20, 2020

Uh oh!

turbaszek Oct 20, 2020

Choose a reason for hiding this comment

Uh oh!

turbaszek Oct 20, 2020

Choose a reason for hiding this comment

Uh oh!

turbaszek commented Oct 20, 2020

Uh oh!

sharanf commented Oct 20, 2020

Uh oh!

turbaszek commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

turbaszek commented Oct 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

turbaszek commented Oct 10, 2020 •

edited

Loading

turbaszek commented Oct 17, 2020 •

edited

Loading

turbaszek commented Oct 20, 2020 •

edited

Loading