Can we not run connectors locally w/o a Fluvio cloud account?

[sean-mca]

running fluvio hub connector download infinyon/[email protected] prompts me to use the CDK to do it instead, which means running cdk hub download infinyon/[email protected]. this gives the following terminal output:

Error: downloading infinyon/sql-sink-0.3.0.ipkg

Server: no access credentials, try fluvio cloud login

I like fluvio as a concept, but I find some of the design choices (like this one, if its intentional) confusing // geared at pushing people to your cloud offering, rather than actually offering an OSS version. having a cloud account to test is a deal breaker at my current role, which is a shame because I really want to use this over kafka

[ajhunyady]

Hi @SeanMcA-EAI, thanks for the feedback. Our next revision of the Cloud scheduled to release in about a month will make the Hub a bit more OSS friendly with the ability to download without authorization.

However, the Hub also allows developers to upload components, which we cannot allow without authorization.

[drc-infinyon]

Edit:
we've removed the account requirements for listing and downloading public connectors and smartmodules. The cli update will release with fluvio 0.11.5 coming up, or you can try it now in the latest channel with fvm install latest.

This should also allow downloading and using ipkg files with fluvio in situations like container building or offline environments.

---

Hey @SeanMcA-EAI - You don't need to use the cloud offering. You need to credentials to download the precompiled hub packages thats all.

That is completely free and does not require you to use our managed offering. It is just a simple way of us knowing that the things on the hub are being downloaded to justify the value of these components.

The SQL connector is open source - https://github.com/infinyon/sql-connector. And CDK (Connector Development Kit) is capable to build the ipkg instead of downloading. So you can clone the repo and build the connector locally. Which is the self hosted experience.

We are delineating the open source and the cloud clearly and you are right there are some confusing elements due to the way the technology is organized. We want a very clear cut self hosted developer experience that is completely separate from the managed cloud experience.

In the OSS experience we have the dev kits to build and compile self hosted open source software. We need credentials for uploading to ensure the overall system's security and prevent malicious intent.

The InfinyOn Cloud Hub is a place where we are hosting compiled binaries and is separate from the Managed Fluvio Clusters as a service. Authorization is required because we also have the capability to upload compiled packages into hub. I should also add that we are working to separate the download and upload flow to require authentication for upload only in the near future.

I hope this offers some clarity and I appreciate your feedback. Please let me know if you have any further questions.

[sean-mca]

@drc-infinyon I hear you, and to be clear I understand from your POV! but, for working on a POC at my current role, I can't make an account with you guys (even if it is free & just to use the hub packages) without going through sourcing, etc.

I understand that's also an operational issue on my side, but I don't think I'm the only one with such requirements, either.

the sql connector might be open source, yea, but to use it, every example so far makes it seem that I need to also use infinyon/json-sql, which I need to access the hub for, no? that's my issue here

[drc-infinyon]

@SeanMcA-EAI this is the price to pay for deconstruction or composability.

Every connector and smart module is open source. It's just not a mono repo.

For the connectors:
Clone repo and use CDK to build

HTTP Source: https://github.com/infinyon/http-source-connector
HTTP Sink: https://github.com/infinyon/http-sink-connector
DuckDB: https://github.com/infinyon/duckdb-connector
Kafka: https://github.com/infinyon/kafka-connector
MQTT: https://github.com/infinyon/mqtt-connector
NATS: https://github.com/infinyon/nats-connector
SQL: https://github.com/infinyon/sql-connector
Graphite: https://github.com/infinyon/graphite-sink-connector

For the smart modules:

Clone repo and use SMDK to build

Labs Projects: https://github.com/infinyon/labs-projects (with connectors and modules)
JSON-SQL: https://github.com/infinyon/sql-connector/tree/main/crates/json-sql
JOLT: https://github.com/infinyon/fluvio-jolt

I will work on cleaning some of this up to help you. Let me know what components you need. Happy to help.

[ajhunyady]

@drc-infinyon I hear you, and to be clear I understand from your POV! but, for working on a POC at my current role, I can't make an account with you guys (even if it is free & just to use the hub packages) without going through sourcing, etc.

I understand that's also an operational issue on my side, but I don't think I'm the only one with such requirements, either.

the sql connector might be open source, yea, but to use it, every example so far makes it seem that I need to also use infinyon/json-sql, which I need to access the hub for, no? that's my issue here

@SeanMcA-EAI , yes, the connector uses the json-sql SmartModule, which is also in the open source (https://github.com/infinyon/sql-connector/tree/main/crates/json-sql) and can be compiled and uploaded to fluvio outside of the Hub.

Are you allowed to join our Discord server? That way, we could help you circumvent these issues so you can proceed with the POC outside of the Hub before the authorization issue is addressed.

[digikata]

Hi @SeanMcA-EAI, just dropping you a note that we've removed the account requirements for listing and downloading public connectors and smartmodules. The cli update will release with fluvio 0.11.5 coming up, or you can try it now in the latest channel with fvm install latest.

This should also allow downloading and using ipkg files with fluvio in situations like container building or offline environments.