Skip to content
This repository has been archived by the owner on Apr 5, 2023. It is now read-only.

Eh main/feature/consumer #18

Merged
merged 112 commits into from
May 4, 2022
Merged
Show file tree
Hide file tree
Changes from 87 commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
f60cb61
feat: add worker class to manager the active job
eHattori Apr 11, 2022
8ad2bc1
feat: add libraries to gemfile
eHattori Apr 11, 2022
642d029
feat: add config to rubocop
eHattori Apr 11, 2022
d9c66b4
feat: add worker to boot app
eHattori Apr 11, 2022
f9be3ab
feat: add default worker options
eHattori Apr 11, 2022
c6b6bba
feat: add self.included to add attr in runtime
eHattori Apr 11, 2022
9597d62
feat: add .env.local to dev enviroment
eHattori Apr 12, 2022
52bdc63
refact: url of queue
eHattori Apr 12, 2022
21b9a52
feat: implement a perform_async method
eHattori Apr 12, 2022
6978417
feat: create a parent broker and impl aws_broker
eHattori Apr 12, 2022
44efd09
test: implement fail to broker
eHattori Apr 13, 2022
80a06e2
feat: add config to broker
eHattori Apr 13, 2022
3f4a254
feat: add resourceError
eHattori Apr 14, 2022
0df13ca
fix: stub problem and configure poller
eHattori Apr 14, 2022
496b17b
feat: a test to aws_broker
eHattori Apr 14, 2022
3d6922d
feat: adjust erros to aws_broker
eHattori Apr 15, 2022
20b03f4
feat: adjust options
eHattori Apr 15, 2022
6a8716f
feat: add test to poller success
eHattori Apr 15, 2022
fc0ec26
refact: options to isolate arguments
eHattori Apr 15, 2022
3de01cf
refact: build_instance_broker
eHattori Apr 15, 2022
2e3ad1f
refact: perform_asyn to process_message
eHattori Apr 15, 2022
45296e5
Ignore log folder
ferreira-mev Apr 18, 2022
ba704e9
Add "spec" file for the logger module
ferreira-mev Apr 18, 2022
a3c4b65
Outline Logging module (shared logger object provider)
ferreira-mev Apr 18, 2022
65b32c6
Format JSON output, improve static method inclusion, require logging …
ferreira-mev Apr 18, 2022
14b84c6
Refactor to include Logging as part of PipefyMessage
ferreira-mev Apr 18, 2022
dd2394e
Include calling obj information on JSON output
ferreira-mev Apr 19, 2022
4deebe1
Add preliminary comments and logs to consumer implementation
ferreira-mev Apr 19, 2022
715bfcb
Rephrase log message
ferreira-mev Apr 19, 2022
6a59709
feat: add worker class to manager the active job
eHattori Apr 11, 2022
4cb8c1c
feat: add libraries to gemfile
eHattori Apr 11, 2022
913f990
feat: add config to rubocop
eHattori Apr 11, 2022
df2b7cf
feat: add worker to boot app
eHattori Apr 11, 2022
e5ff577
feat: add default worker options
eHattori Apr 11, 2022
b40d85f
feat: add self.included to add attr in runtime
eHattori Apr 11, 2022
cb790c3
feat: add .env.local to dev enviroment
eHattori Apr 12, 2022
f2cf8a0
refact: url of queue
eHattori Apr 12, 2022
7c82f84
feat: implement a perform_async method
eHattori Apr 12, 2022
1045cf4
feat: create a parent broker and impl aws_broker
eHattori Apr 12, 2022
8155d8c
test: implement fail to broker
eHattori Apr 13, 2022
c7e2dd6
feat: add config to broker
eHattori Apr 13, 2022
109a8f3
feat: add resourceError
eHattori Apr 14, 2022
c5ba545
fix: stub problem and configure poller
eHattori Apr 14, 2022
0b1db3f
feat: a test to aws_broker
eHattori Apr 14, 2022
4d7dd50
feat: adjust erros to aws_broker
eHattori Apr 15, 2022
374fa3e
feat: adjust options
eHattori Apr 15, 2022
7f68344
feat: add test to poller success
eHattori Apr 15, 2022
40b85ab
refact: options to isolate arguments
eHattori Apr 15, 2022
eeac6a2
refact: build_instance_broker
eHattori Apr 15, 2022
dc6f246
refact: perform_asyn to process_message
eHattori Apr 15, 2022
9eb73b4
Merge branch 'eh-main/feature/consumer' of github.com:pipefy/pipefy_m…
eHattori Apr 19, 2022
0e2385f
Merge CustomLogger class config into Logging module
ferreira-mev Apr 19, 2022
37f84c6
refact: remove beta consumer version
eHattori Apr 19, 2022
8bd7111
Use Logging module in pipefy_message.rb
ferreira-mev Apr 19, 2022
0ed0fd6
Merge Hattori and Duda's changes (formatting logs, removing tests out…
ferreira-mev Apr 19, 2022
d2f2c5c
refact: logs and elapsed time to ensure
eHattori Apr 19, 2022
52b4d73
refact: remove JSON.parse
eHattori Apr 19, 2022
746f385
Refactor publisher to incorporate new centralized logger instance
ferreira-mev Apr 20, 2022
890bce8
Remove old logger
ferreira-mev Apr 25, 2022
7f38b09
refact: namespaces
eHattori Apr 25, 2022
155f288
refact: rename publisher class
eHattori Apr 25, 2022
a7a32b5
refact: move base_publisher to publisher
eHattori Apr 25, 2022
183b6ce
refact: create instance by inflection
eHattori Apr 25, 2022
91a4cc6
refact: fix "aws" and worker feature
eHattori Apr 25, 2022
6952468
refact: Move common AWS broker methods to superclass
ferreira-mev Apr 26, 2022
f8f5be6
refact: Document AWS broker class, move shared imports
ferreira-mev Apr 26, 2022
af41655
"fix": Refact SQS err to SNS err on SNS rescue
ferreira-mev Apr 26, 2022
b767f61
refact: Open AWS connection on superclass constructor
ferreira-mev Apr 26, 2022
a21292a
refact: Rewrite publisher logs in JSON format
ferreira-mev Apr 26, 2022
5309c6b
refact: configuration
eHattori Apr 26, 2022
e24bfc1
Documenting SNS broker (partial commit before pulling)
ferreira-mev Apr 26, 2022
f508676
Merge branch 'eh-main/feature/consumer' of github.com:pipefy/pipefy_m…
ferreira-mev Apr 26, 2022
9c8ceee
refact: Remove old AWS configuration class
ferreira-mev Apr 26, 2022
3e454cf
Move SQS, SNS README to AWS client dir
ferreira-mev Apr 26, 2022
fe59799
Handle invalid provider option
ferreira-mev Apr 26, 2022
54dec70
refact: Use described_class rather than hardcoded class name on RSpec…
ferreira-mev Apr 26, 2022
39263ce
refact: Specify which exceptions to handle (WIP - should be improved?)
ferreira-mev Apr 26, 2022
f34f2b0
refact: Separate contexts in worker tests
ferreira-mev Apr 26, 2022
55b1c56
Try to improve worker tests (keyword "try" :P)
ferreira-mev Apr 26, 2022
f654212
refact: Address Rubocop offenses
ferreira-mev Apr 27, 2022
e05ea12
refact: Incorporate review feedback into Logging module
ferreira-mev Apr 27, 2022
aad3ff4
refact: (post-review) Disable AbcSize, MethodLength globally
ferreira-mev Apr 27, 2022
597f7c6
feat: (post-review) Create module to autolog all custom errors
ferreira-mev Apr 27, 2022
b0ca1d7
refact (minor): Please Rubocop
ferreira-mev Apr 27, 2022
f856b3a
refact: (post-review) Move class_path hash to BrokerResolver class
ferreira-mev Apr 27, 2022
7317f1d
refact: Add... missing trailing newline. Yeah. :P
ferreira-mev Apr 27, 2022
788e307
refact: (post-review) Move SQS-specific poller wait time option to Sq…
ferreira-mev Apr 27, 2022
23cb031
refact (post-review): Remove logger methods made unnecessarily verbos…
ferreira-mev Apr 28, 2022
92dd3cc
refact: Update log message to reflect ms time unit
ferreira-mev Apr 28, 2022
7badbfe
refact: [WIP?] Improve separation of AWS-wide and service-specific co…
ferreira-mev Apr 28, 2022
9d08224
refact: Wait longer for localstack init (10 s wasn't enough on my mac…
ferreira-mev Apr 29, 2022
0da3c7f
refact: Add queue name to logs if not already present
ferreira-mev Apr 29, 2022
5298582
refact: Reword/break lines
ferreira-mev Apr 29, 2022
02d097d
refact: Expand AWS SQS, SNS CLI readme
ferreira-mev Apr 29, 2022
4faaf42
refact/fix: Alter test to reflect new default arg value
ferreira-mev Apr 29, 2022
9cc0163
refact: Address some minor issues from the PR review
ferreira-mev Apr 29, 2022
bd1af3a
Remove AwsBroker -> {Sns, Sqs}Broker inheritance
ferreira-mev May 2, 2022
2be266a
refact: Remove currently unused Broker class
ferreira-mev May 2, 2022
d63ca52
(forgot to include all files in the last commit, then pushed :P)
ferreira-mev May 2, 2022
c53ff12
feat/refact?: Allow setting broker options from Worker module and Pub…
ferreira-mev May 2, 2022
253dd05
Add and improve RSpec tests
ferreira-mev May 3, 2022
b2f8d9c
[WIP] Polish
jleber May 3, 2022
6716678
refact: Refactor tests to match lib folder refactor
ferreira-mev May 3, 2022
b59e4f9
Merge pull request #19 from pipefy/ef/refactor/remove-inheritance
ferreira-mev May 4, 2022
2809815
Update project documentation
jleber May 4, 2022
12b6293
Adjust Aws Client
jleber May 4, 2022
1860bcf
Fix for log separator
jleber May 4, 2022
67a2a42
Adjust consumer stuff
jleber May 4, 2022
ecacdb1
Adjust log message and add sample consumer class
jleber May 4, 2022
58f40d6
Linter adjustments
jleber May 4, 2022
6d0f63d
refact (minor): Nitpicky uniformization of || vs fetch
ferreira-mev May 4, 2022
dbe13de
Quick README review w/ minor grammar-related edits and clarifications
ferreira-mev May 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.local
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
export AWS_ACCESS_KEY_ID=foo
export AWS_SECRET_ACCESS_KEY=bar
export AWS_ENDPOINT="http://localhost:4566"
export ENABLE_AWS_CLIENT_CONFIG=true
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,5 @@ build-iPhoneSimulator/

.idea
.rspec_status

ferreira-mev marked this conversation as resolved.
Show resolved Hide resolved
/log/
20 changes: 19 additions & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
AllCops:
TargetRubyVersion: 2.6
NewCops: enable

Style/StringLiterals:
Enabled: true
Expand All @@ -13,4 +14,21 @@ Layout/LineLength:
Max: 120

Metrics/BlockLength:
IgnoredMethods: ['describe', 'context']
IgnoredMethods: ["describe", "context"]

Style/GlobalVars:
Exclude:
- "spec/worker_spec.rb"
# The introduction of a dummy global variable greatly simplified
# testing in this case, as it made reassignment between scopes much
# more straightforward. I didn't consider that particularly harmful.
# Thoughts?
ferreira-mev marked this conversation as resolved.
Show resolved Hide resolved

Lint/MissingSuper:
Enabled: false

Metrics/MethodLength:
Enabled: false

Metrics/AbcSize:
Enabled: false
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"rufo.useBundler": false,
}
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ group :development, :test do
gem "pry-doc"
gem "rspec", "~> 3.0"
gem "rubocop", "~> 1.21"
gem "solargraph"
end
33 changes: 33 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,27 @@ GEM
aws-sigv4 (~> 1.1)
aws-sigv4 (1.4.0)
aws-eventstream (~> 1, >= 1.0.2)
backport (1.2.0)
benchmark (0.2.0)
coderay (1.1.3)
concurrent-ruby (1.1.9)
diff-lcs (1.3)
e2mmap (0.1.0)
i18n (1.10.0)
concurrent-ruby (~> 1.0)
jaro_winkler (1.5.4)
jmespath (1.5.0)
kramdown (2.3.2)
rexml
kramdown-parser-gfm (1.1.0)
kramdown (~> 2.0)
logger (1.3.0)
method_source (1.0.0)
minitest (5.15.0)
nokogiri (1.13.3-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.13.3-x86_64-linux)
racc (~> 1.4)
parallel (1.21.0)
parser (3.0.2.0)
ast (~> 2.4.1)
Expand All @@ -47,9 +59,12 @@ GEM
pry-doc (1.3.0)
pry (~> 0.11)
yard (~> 0.9.11)
racc (1.6.0)
rainbow (3.0.0)
rake (13.0.6)
regexp_parser (2.1.1)
reverse_markdown (2.1.1)
nokogiri
rexml (3.2.5)
rspec (3.8.0)
rspec-core (~> 3.8.0)
Expand All @@ -76,7 +91,23 @@ GEM
rubocop-ast (1.12.0)
parser (>= 3.0.1.1)
ruby-progressbar (1.11.0)
solargraph (0.41.2)
backport (~> 1.1)
benchmark
bundler (>= 1.17.2)
e2mmap
jaro_winkler (~> 1.5)
kramdown (~> 2.3)
kramdown-parser-gfm (~> 1.1)
parser (~> 3.0)
reverse_markdown (>= 1.0.5, < 3)
rubocop (>= 0.52)
thor (~> 1.0)
tilt (~> 2.0)
yard (~> 0.9, >= 0.9.24)
thor (1.2.1)
thread_safe (0.3.6)
tilt (2.0.10)
tzinfo (1.2.9)
thread_safe (~> 0.1)
unicode-display_width (2.1.0)
Expand All @@ -85,6 +116,7 @@ GEM
webrick (~> 1.7.0)

PLATFORMS
ruby
x86_64-darwin-20
x86_64-linux

Expand All @@ -99,6 +131,7 @@ DEPENDENCIES
rake (~> 13.0)
rspec (~> 3.0)
rubocop (~> 1.21)
solargraph

BUNDLED WITH
2.3.9
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,22 @@ On the irb console:

* Consume a message
```ruby
require 'pipefy_message'
message = PipefyMessage::Test.new
message.consume
require "pipefy_message"

class TestWorker
include PipefyMessage::Worker
pipefymessage_options broker: "aws", queue_name: "pipefy-local-queue"

def perform(message)
puts message
end
end

TestWorker.perform_async
```



* Publish and Consume a message
```ruby
require 'pipefy_message'
Expand All @@ -82,7 +93,7 @@ On the irb console:

## Project Stack

- [Aws SDK Ruby - SNS & SQS](https://github.com/aws/aws-sdk-ruby)
- [Aws SDK Ruby - SNS & SQS](https://github.com/aws_client/aws-sdk-ruby)
- [Bundler](https://bundler.io/)
- Docker-compose
- [GitHub Actions](https://docs.github.com/en/actions)
Expand All @@ -92,7 +103,7 @@ On the irb console:

## Brokers Documentation

* [SNS & SQS User guide](https://github.com/pipefy/pipefy_message/tree/main/lib/pipefy_message/broker/aws/README.md)
* [SNS & SQS User guide](https://github.com/pipefy/pipefy_message/tree/main/lib/pipefy_message/broker/aws_client/README.md)

## Contributing

Expand Down
41 changes: 13 additions & 28 deletions lib/pipefy_message.rb
Original file line number Diff line number Diff line change
@@ -1,35 +1,20 @@
# frozen_string_literal: true

require_relative "pipefy_message/version"
require_relative "pipefy_message/broker/aws/configuration"
require_relative "pipefy_message/broker/aws/sns/publisher"
require_relative "pipefy_message/base_consumer"
require_relative "pipefy_message/base_publisher"
require_relative "pipefy_message/logger"
require_relative "pipefy_message/logging"
require_relative "pipefy_message/publisher"
require_relative "pipefy_message/worker"
require_relative "pipefy_message/providers/broker"
require_relative "pipefy_message/providers/errors"

require "logger"
require "json"
require "benchmark"
require "active_support"
require "active_support/core_ext/string/inflections"

##
# PipefyMessage abstraction async process
##
module PipefyMessage
# Simple Test class to validate the project
class Test
def initialize
@log = PipefyMessage::CustomLogger.new
end

def publish
payload = { foo: "bar" }
puts Publisher::BasePublisher.new.publish(payload, "pipefy-local-topic")
end

def consume
@log.info("Starting the consumer process")
consumer = BaseConsumer.new("http://localhost:4566/000000000000/pipefy-local-queue")
@log.info("Creating new instance of consumer #{consumer}")
consumer.consume_message
end

def publish_and_consume
publish
consume
end
end
end
15 changes: 0 additions & 15 deletions lib/pipefy_message/base_consumer.rb

This file was deleted.

31 changes: 0 additions & 31 deletions lib/pipefy_message/base_publisher.rb

This file was deleted.

39 changes: 0 additions & 39 deletions lib/pipefy_message/broker/aws/configuration.rb

This file was deleted.

51 changes: 0 additions & 51 deletions lib/pipefy_message/broker/aws/sns/publisher.rb

This file was deleted.

Loading