Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NUTCH-2856] Implement a protocol-smb plugin based on hierynomus/smbj #826

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

[NUTCH-2856] Implement a protocol-smb plugin based on hierynomus/smbj #826

wants to merge 10 commits into from

Conversation

HiranChaudhuri
Copy link
Contributor

[NUTCH-2856] Implement a protocol-smb plugin based on hierynomus/smbj

Draft version of a protocol-smb plugin. Lots of todo comments still,
but it seems to work.

[NUTCH-2856] Implement a protocol-smb plugin based on hierynomus/smbj
b568cc9 
Draft version of a protocol-smb plugin. Lots of todo comments still,
but it seems to work.
@lewismc lewismc marked this pull request as draft October 3, 2024 04:10
@lewismc
Copy link
Member

lewismc commented Oct 3, 2024

Moving this to DRAFT status and acknowledging the PR @HiranChaudhuri thank you.
I will try to perform a full review this coming week... thank you.

@lewismc lewismc mentioned this pull request Oct 4, 2024
Merged
lewismc
lewismc requested changes Oct 4, 2024
View reviewed changes
Copy link
Member

@lewismc lewismc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @HiranChaudhuri I added quite a few comments for your consideration. Thanks for submitting this PR 👍
Please ping me once your ready and we can go for round # 2 of peer review.

Further out, I think we could implement some testing for this protocol plugin. We could use testcontainers and essentially spin up a local Samba server using @nddipiazza 's smbj-inttest image. We can come back to this one the PR has evolved a bit.

.gitignore Outdated Show resolved Hide resolved
conf/log4j2.xml Show resolved Hide resolved
runNutch.sh Outdated Show resolved Hide resolved
src/plugin/build.xml Show resolved Hide resolved
src/plugin/protocol-smb/ivy.xml Show resolved Hide resolved
src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/Smb.java Outdated Show resolved Hide resolved
src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/Smb.java Outdated Show resolved Hide resolved
src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/Smb.java Outdated Show resolved Hide resolved
src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/Smb.java Outdated Show resolved Hide resolved
src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/SmbURLConnection.java Show resolved Hide resolved
Reduce logging
a5ad1da 
Improve error handling
Rename class as requested
Added license header
Improve url parsing
added robots.txt
lewismc
lewismc requested changes Oct 14, 2024
View reviewed changes
Copy link
Member

@lewismc lewismc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all of the responses @HiranChaudhuri please see my updated suggestions.

.gitignore Outdated Show resolved Hide resolved
conf/log4j2.xml Show resolved Hide resolved
runNutch.sh Outdated Show resolved Hide resolved
src/plugin/build.xml Show resolved Hide resolved
src/plugin/protocol-smb/ivy.xml Show resolved Hide resolved
src/plugin/protocol-smb/ivy.xml Show resolved Hide resolved
src/plugin/protocol-smb/plugin.xml Show resolved Hide resolved
@lewismc
Copy link
Member

lewismc commented Oct 16, 2024

Hi @HiranChaudhuri now that you've activated the plugin test target, I suggest that we implement the unit tests using the test-containers utility. An example can be found at https://java.testcontainers.org/quickstart/junit_4_quickstart/.
For example we could use

@Rule
public GenericContainer redis = new GenericContainer(DockerImageName.parse("ndipiazza/smbj-inttest:1.1"))
    .withExposedPorts(445);

Please let me know your thoughts on this.

@HiranChaudhuri
Copy link
Contributor Author

The container looks good. I have no clue about the @rule annotation and am interested to see how this gets combined together.

lewismc
lewismc requested changes Oct 16, 2024
View reviewed changes
Copy link
Member

@lewismc lewismc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial header requests. Thanks

conf/url-authentication.xml.template
@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide ALv2 header

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

src/plugin/protocol-smb/src/java/org/apache/nutch/protocol/smb/URLAuthentication.java
@@ -0,0 +1,120 @@
package org.apache.nutch.protocol.smb;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide ALv2 header

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@HiranChaudhuri HiranChaudhuri marked this pull request as ready for review October 18, 2024 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lewismc lewismc lewismc requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@HiranChaudhuri @lewismc