Skip to content

Commit

Permalink
docker-compose changes (#764) (#765)
Browse files Browse the repository at this point in the history 
* docker-compose changes (#764)

revert kudu change

Update check-build-test.yml

* copy backup couchbase docker setup
  • Loading branch information
@pjfanning
pjfanning authored Aug 7, 2024
1 parent 56e1f56 commit 27822fb
Show file tree
Hide file tree
Showing 8 changed files with 66 additions and 59 deletions.
43 changes: 22 additions & 21 deletions .github/workflows/check-build-test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,49 +76,50 @@ jobs:
fail-fast: false
matrix:
include:
- { connector: amqp, pre_cmd: 'docker-compose up -d amqp' }
- { connector: amqp, pre_cmd: 'docker compose up -d amqp' }
- { connector: avroparquet }
- { connector: awslambda }
- { connector: aws-event-bridge, pre_cmd: 'docker-compose up -d amazoneventbridge' }
- { connector: aws-event-bridge, pre_cmd: 'docker compose up -d amazoneventbridge' }
- { connector: azure-storage-queue }
- { connector: cassandra, pre_cmd: 'docker-compose up -d cassandra' }
- { connector: couchbase, pre_cmd: 'docker-compose up -d couchbase_prep' }
- { connector: cassandra, pre_cmd: 'docker compose up -d cassandra' }
- { connector: couchbase, pre_cmd: 'docker compose up -d couchbase_prep' }
- { connector: csv }
- { connector: dynamodb, pre_cmd: 'docker-compose up -d dynamodb' }
- { connector: elasticsearch, pre_cmd: 'docker-compose up -d elasticsearch6 elasticsearch7 opensearch1' }
- { connector: dynamodb, pre_cmd: 'docker compose up -d dynamodb' }
- { connector: elasticsearch, pre_cmd: 'docker compose up -d elasticsearch6 elasticsearch7 opensearch1' }
- { connector: file }
- { connector: ftp, pre_cmd: './scripts/ftp-servers.sh' }
- { connector: geode, pre_cmd: 'docker-compose up -d geode' }
- { connector: geode, pre_cmd: 'docker compose up -d geode' }
- { connector: google-cloud-bigquery }
- { connector: google-cloud-bigquery-storage }
- { connector: google-cloud-pub-sub, pre_cmd: 'docker-compose up -d gcloud-pubsub-emulator_prep' }
- { connector: google-cloud-pub-sub-grpc, pre_cmd: 'docker-compose up -d gcloud-pubsub-emulator_prep' }
- { connector: google-cloud-pub-sub, pre_cmd: 'docker compose up -d gcloud-pubsub-emulator_prep' }
- { connector: google-cloud-pub-sub-grpc, pre_cmd: 'docker compose up -d gcloud-pubsub-emulator_prep' }
- { connector: google-cloud-storage }
- { connector: google-common }
- { connector: google-fcm }
# hbase disabled until we resolve why new docker image fails our build: https://github.com/akka/alpakka/issues/2185
# - { connector: hbase, pre_cmd: 'docker-compose up -d hbase' }
# - { connector: hbase, pre_cmd: 'docker compose up -d hbase' }
- { connector: hdfs, pre_cmd: 'file ${HOME}/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_2.13/2.0.0/cats-kernel_2.13-2.0.0.jar' }
- { connector: huawei-push-kit }
- { connector: influxdb, pre_cmd: 'docker-compose up -d influxdb' }
- { connector: ironmq, pre_cmd: 'docker-compose up -d ironauth ironmq' }
- { connector: jms, pre_cmd: 'docker-compose up -d ibmmq' }
- { connector: influxdb, pre_cmd: 'docker compose up -d influxdb' }
# ironmq disabled while we resolve https://github.com/apache/pekko-connectors/issues/697
# - { connector: ironmq, pre_cmd: 'docker compose up -d ironauth ironmq' }
- { connector: jms, pre_cmd: 'docker compose up -d ibmmq' }
- { connector: json-streaming }
- { connector: kinesis }
- { connector: kudu, pre_cmd: 'docker-compose up -d kudu-master-data kudu-tserver-data kudu-master kudu-tserver' }
- { connector: mongodb, pre_cmd: 'docker-compose up -d mongo' }
- { connector: mqtt, pre_cmd: 'docker-compose up -d mqtt' }
- { connector: mqtt-streaming, pre_cmd: 'docker-compose up -d mqtt' }
- { connector: orientdb, pre_cmd: 'docker-compose up -d orientdb' }
- { connector: pravega, pre_cmd: 'docker-compose up -d pravega'}
- { connector: kudu, pre_cmd: 'docker compose up -d kudu-master-data kudu-tserver-data kudu-master kudu-tserver' }
- { connector: mongodb, pre_cmd: 'docker compose up -d mongo' }
- { connector: mqtt, pre_cmd: 'docker compose up -d mqtt' }
- { connector: mqtt-streaming, pre_cmd: 'docker compose up -d mqtt' }
- { connector: orientdb, pre_cmd: 'docker compose up -d orientdb' }
- { connector: pravega, pre_cmd: 'docker compose up -d pravega'}
- { connector: reference }
- { connector: s3 }
- { connector: spring-web }
- { connector: simple-codecs }
- { connector: slick }
- { connector: sns, pre_cmd: 'docker-compose up -d amazonsns' }
- { connector: sns, pre_cmd: 'docker compose up -d amazonsns' }
- { connector: solr }
- { connector: sqs, pre_cmd: 'docker-compose up -d elasticmq' }
- { connector: sqs, pre_cmd: 'docker compose up -d elasticmq' }
- { connector: sse }
- { connector: text }
- { connector: udp }
Expand Down
2 changes: 1 addition & 1 deletion build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ lazy val `pekko-connectors` = project
| starts a webserver and opens a new browser window
|
| test - runs all the tests for all of the connectors.
| Make sure to run `docker-compose up` first.
| Make sure to run `docker compose up` first.
|
| mqtt/testOnly *.MqttSourceSpec - runs a single test
|
Expand Down
70 changes: 38 additions & 32 deletions docker-compose.yml
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# For detailed information about docker-compose visit https://docs.docker.com/compose/
# To start all docker containers required to execute the tests locally run:
# docker-compose up
# docker compose up

version: '2.2'
services:
amazoneventbridge:
image: localstack/localstack:0.10.6
Expand All @@ -26,44 +25,51 @@ services:
ports:
- "9042:9042"
couchbase:
image: couchbase:community-5.1.1
image: couchbase:community-7.6.1
ports:
- "8091-8094:8091-8094"
- "11210:11210"
- "8091-8096:8091-8096"
- "11210-11211:11210-11211"
couchbase_prep:
image: couchbase:community-5.1.1
image: couchbase:community-7.6.1
links:
- "couchbase"
entrypoint: ""
# when we drop couchbase2, we don't need to create bucket pekkoquery
command: >
bash -c "
echo 'waiting until couchbase is up'
until `curl --output /dev/null --silent --head --fail http://couchbase:8091`; do
bash -c
"echo 'waiting until couchbase is up'
until `curl --output /dev/null --silent --head --fail http://couchbase:8091`; do
printf '.'
sleep 2
done
couchbase-cli cluster-init -c couchbase \
--cluster-username Administrator --cluster-password password \
--cluster-ramsize 300 \
--cluster-index-ramsize 256 \
--services data,index,query,fts
couchbase-cli bucket-create -c couchbase \
-u Administrator -p password \
--bucket pekko \
--bucket-type couchbase \
--bucket-ramsize 100 \
--bucket-replica 1 \
--wait
couchbase-cli bucket-create -c couchbase \
-u Administrator -p password \
--bucket pekkoquery \
--bucket-type couchbase \
--bucket-ramsize 100 \
--bucket-replica 1 \
--wait
sleep 2 # just wait a tiny bit more after creating the bucket
echo 'CREATE PRIMARY INDEX ON pekkoquery USING GSI;' | \
cbq -c Administrator:password -e http://couchbase:8093
done
couchbase-cli cluster-init -c couchbase
--cluster-username Administrator
--cluster-password password
--cluster-ramsize 300
--cluster-index-ramsize 256
--cluster-fts-ramsize 256
--cluster-query-ramsize 256
--services data,index,query,fts
couchbase-cli bucket-create -c couchbase
-u Administrator -p password
--bucket pekko
--bucket-type couchbase
--bucket-ramsize 120
--bucket-replica 1
--wait
couchbase-cli bucket-create -c couchbase
-u Administrator -p password
--bucket pekkoquery
--bucket-type couchbase
--bucket-ramsize 120
--bucket-replica 1
--wait
sleep 2 # just wait a tiny bit more after creating the bucket
"
elasticmq:
image: softwaremill/elasticmq-native:1.3.4
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/paradox/aws-event-bridge.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ Messages published in a batch using @apidoc[EventBridgePublisher.flowSeq](EventB

## Integration testing

For integration testing without connecting directly to Amazon EventBridge, Apache Pekko Connectors uses [Localstack](https://github.com/localstack/localstack), which comes as a docker image - and has a corresponding service `amazoneventbridge` in the `docker-compose.yml` file. Which needs to be started before running the integration tests `docker-compose up amazoneventbridge`.
For integration testing without connecting directly to Amazon EventBridge, Apache Pekko Connectors uses [Localstack](https://github.com/localstack/localstack), which comes as a docker image - and has a corresponding service `amazoneventbridge` in the `docker-compose.yml` file. Which needs to be started before running the integration tests `docker compose up amazoneventbridge`.

@@@ index

Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/paradox/ftp.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Java
The code in this guide is part of runnable tests of this project. You are welcome to browse the code, edit and run it in sbt.

```
docker-compose up -d ftp sftp
docker compose up -d ftp sftp
sbt
> ftp/test
```
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/paradox/google-cloud-pub-sub-grpc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ Java
@@@ note
Integration test code requires Google Cloud Pub/Sub emulator running in the background. You can start it quickly using docker:

`docker-compose up -d gcloud-pubsub-client`
`docker compose up -d gcloud-pubsub-client`

This will also run the Pub/Sub admin client that will create topics and subscriptions used by the
integration tests.
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/paradox/mqtt.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ The code in this guide is part of runnable tests of this project. You are welcom

> Test code requires a MQTT server running in the background. You can start one quickly using docker:
>
> `docker-compose up mqtt`
> `docker compose up mqtt`
Scala
: ```
Expand Down
2 changes: 1 addition & 1 deletion scripts/ftp-servers.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ set -x
export FTP_USER_UID=$(id -u)
export FTP_USER_GID=$(id -g)

docker-compose up -d ftp sftp squid
docker compose up -d ftp sftp squid

0 comments on commit 27822fb

Please sign in to comment.