fetch specs from definitions directly #7293
Conversation
lmossman
changed the title
|
I'm not sure why the Build steps are failing, both the Connectors Base and Platform builds finish successfully for me locally. Looks like the Platform build failure is related to this: which looks like it is also occurring in this PR: #7288 (comment) |
| // and | ||
| // specific to a job id, allowing us to do this. | ||
| // attempts ids are monotonically increasing starting from | ||
| // 0 and specific to a job id, allowing us to do this. |
There was a problem hiding this comment.
Nit: merge the 2 comment lines.
| final SynchronousResponse<ConnectorSpecification> response = specFetcher.getSpecJobResponse(sourceDefinition); | ||
|
|
||
| assertEquals(ConfigType.GET_SPEC, response.getMetadata().getConfigType()); | ||
| assertEquals(Optional.empty(), response.getMetadata().getConfigId()); |
There was a problem hiding this comment.
It is really up to you especially on an existing test, I don't want to sound too pushy on that. You could use assertJ here Assertions.assertThat(response.getMetadata().getConfigId()).isEmpty()
There was a problem hiding this comment.
@benmoriceau Does using assertJ here result in any meaningful benefit? I'm a little bit hesitant to change this test to use assertJ because the jupiter assertions are used across all of our other tests, and it seems slightly worse to me to use 2 different assertion libraries across the codebase.
I think if we make a decision as an org to switch to using assertJ going forward, then I would be happy to make this change. Maybe we can try to discuss it during a company meeting?
There was a problem hiding this comment.
There is a clear benefits in testing exception. It also provides a better display and interface for build-in types such as Optional or collection. This article show some example of the benefits: https://annaduldiier.medium.com/assertj-vs-junit-483b7d6dc997 even if some part of the article are discutable (like the auto-completion).
I don't think that having 2 assertions framework is an issue especially since in this case the static method are different, there is a very small probability of mismatching one for another one.
There was a problem hiding this comment.
@benmoriceau this seems worth bringing up with the team in the weekly platform sync! help familiarize people with it and get some buy in!
| now, | ||
| now, | ||
| true, | ||
| null); |
There was a problem hiding this comment.
Nit: could you add named variable for thenull and true value, it makes it easier to review.
There was a problem hiding this comment.
Added these named variables to the new static method that I added to the SynchronousJobMetadata class for generating this mocked object 👍
airbyte-server/src/main/java/io/airbyte/server/converters/SpecFetcher.java
Outdated
Show resolved
Hide resolved
| return getSpecFromJob(getSpecJobResponse(destinationDefinition)); | ||
| } | ||
|
|
||
| public SynchronousResponse<ConnectorSpecification> getSpecJobResponse(final StandardSourceDefinition sourceDefinition) throws IOException { |
There was a problem hiding this comment.
how annoying! I see that usage in SchedulerHandler that is causing this. can you add a todo that says when we have moved the spec into the db as a required field we should get rid of the need for this?
in case it's not clear, all jobs the require spinning up a docker container we use this SynchronousResponse struct to help pass through metadata around what happened in the job and logs. Once we can guarantee that for spec that we will not be spinning up a docker container, we can just remove this part from the code path.
There was a problem hiding this comment.
As you pointed out, the main reason I added this was because the SourceDefinitionSpecificationRead struct returned by the SchedulerHandler method (which is in turned returned by the ConfigurationApi) contains a required jobInfo field, so the method I added mocked out that job info when just getting the spec from the db.
Therefore it seems to me that as part of the change to make the spec field required/guaranteed on the db struct, we should also remove the jobInfo field from the SourceDefinitionSpecificationRead struct. Does this align with what you are suggesting here?
There was a problem hiding this comment.
i actually think the code you've written here is good. i'm not suggesting we change anything. just want to add a comment saying that we want to get rid of this part of the public interface in the future.
you're right, we will want to remove jobinfo from SourceDefinitionSpecificationRead, but we should wait to do it until we have the guarantee that we will only ever fetch the spec from the db. the reality is that while it is still possible to pull specs from the docker image, it is still helpful to keep the jobinfo because it includes logs to give clues as to what the problems are!
There was a problem hiding this comment.
Completely agree! I meant that we should only remove jobInfo once we have that guarantee, and I agree that it makes sense to keep it for now 👍
airbyte-server/src/main/java/io/airbyte/server/converters/SpecFetcher.java
Outdated
Show resolved
Hide resolved
airbyte-server/src/test/java/io/airbyte/server/converters/SpecFetcherTest.java
Outdated
Show resolved
Hide resolved
| private final SynchronousSchedulerClient schedulerJobClient; | ||
|
|
||
| public SpecFetcher(final SynchronousSchedulerClient schedulerJobClient) { | ||
| this.schedulerJobClient = schedulerJobClient; | ||
| } | ||
|
|
||
| public ConnectorSpecification execute(final String dockerImage) throws IOException { | ||
| public ConnectorSpecification getSpec(final String dockerImage) throws IOException { |
There was a problem hiding this comment.
it looks like the only reason we need to keep this method are 2 usages.
ServerApp.javaline 222, which we know we are going to get rid of when we get rid of the file migrationsDockerImageValidator- left a comment below for how we might be able to remove this usage.
can we add a todo, reminding us that we want to kill this method?
There was a problem hiding this comment.
the reason i'm so focused on this method is because it has a different behavior that is non-obvious from the caller's point of view. the other methods all will try to find the spec in the definition itself. this one will never do that and will always start calling scheduler clients. so i want to advertise loudly that it is different and shouldn't be used. in fact, we should mark it as deprecated.
airbyte-server/src/main/java/io/airbyte/server/validators/DockerImageValidator.java
Outdated
Show resolved
Hide resolved
|
Oh. as for the build, I'm a little unsure what's going on. It looks like it's failing on something related to python. I just merged a PR that should totally remove the need for the platform build to need python at all. So maybe try rebasing on master and seeing if that helps? I'd like to avoid python oddities if we can just make them irrelevant instead 😉 |
cgardens
added
area/platform
lmossman
force-pushed
the
lmossman/fetch-specs-from-definitions
branch
from
57e5316
to
b4438be
Compare
* try fetching specs from definitions first * refactor specFetcher and update tests * run gradle format * format again * fix comment formatting * fix test * merge comment lines into single line * move duplicate job metadata mocking logic to shared static method * add todo * formatting * use local var and clone * run gw format * add todo * skip spec fetcher in docker image validator and update todos * run gw format
What
Resolves #7136, though a slightly different approach was taken than what was described in the issue, as explained below.
Refactors the SpecFetcher to take in Source and Destination Definitions and attempt to fetch the spec from those structs directly, before falling back to the scheduler job client.
How
Source and Destination definitions are almost always retrieved from the database before the spec fetcher is called, so this change takes advantage of that behavior by allowing those definitions to be passed into the spec fetcher, where it attempts to read the spec from them directly.
Another advantage of this is that the logic for resolving the docker image name from a source/destination definition is now centralized in the spec fetcher and is only performed when the definition doesn't contain a spec, reducing some repeated code.
Recommended reading order
SpecFetcher.javaPre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/SUMMARY.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereUpdating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereConnector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes