Use prepared statements instead of string concatenated SQL everywhere - PART 1 (FINERACT-854)

[josemakara2]

Description

Replace SQL query concatenations with JDBC prepared statements with ? parameters in the code base.
Work done

1. Fixed Parameter Fineract-Platform-TenantId SQL Injection appearing in 29 places on OWASP ZAP scan
   This is fixed in 2 places:- BasicAuthTenantDetailsServiceJdbc.java and JdbcTenantDetailsService.java

> High	SQL Injection
Description	SQL injection may be possible.
URL	https://localhost:8443/fineract-provider/api/v1/users
Method	GET
Parameter	Fineract-Platform-TenantId
Attack	default%

2. Added index identifier to table tenants, related to 1 above. Query no longer appearing in MySQL slow log
3. Fixed REGEX workaround to runreports SQL Injections reported in OWASP ZAP scan see ReadReportingServiceImpl.java
4. When I went to test reports, I realised they were not working. Did history check on github/jira and found FINERATC-1336, FINERATC-1338
   and ended up fixing that in RunreportsApiResource.java
5. 3 other SQLi items in ZAP scan

Next I plan to open 2 more pull requests

1. Fix SQLi issues in the following areas remaining 11 areas mentioned in FINERACT-969 (sub-tasks in FINERATC-854)

• SearchReadPlatformServiceImpl.java
• ReadSurveyServiceImpl.java
• XBRLResultServiceImpl.java
• ProvisioningCriteriaReadPlatformServiceImpl.java
• CashierTransactionDataValidator.java
• AuditReadPlatformServiceImpl.java
• GenericDataServiceImpl.java
• ExternalServicesReadPlatformServiceImpl.java
• ReadWriteNonCoreDataServiceImpl.java
• SmsReadPlatformServiceImpl.java
• ReportMailingJobReadPlatformServiceImpl.java

2. Add ZAP in CI process leveraging ZAP command line interface. Including ZAP in the equation, we will be able to get test and vulnerability assessment results at the same time when the build process is done. We can start by just turning on SQL Injection in ZAP policy and in future we enable full ZAP security scan = more work. Jira FINERATC-969
3. Let me know if you have an alternative approach we can consider.

Checklist

Please make sure these boxes are checked before submitting your pull request - thanks!

• Write the commit message as per https://github.com/apache/fineract/#pull-requests

• Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

• Create/update unit or integration tests for verifying the changes made.

• Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

• Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/api-docs/apiLive.htm with details of any API changes

• Submission is not a "code dump". (Large changes can be made "in repository" via a branch. Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

[thesmallstar]

@josemakara2 hey, thank you so much for the PR.
Can you please update the Description?
Also, I can review the PR once the tests pass.
-Thanks
Manthan

[josemakara2]

@josemakara2 hey, thank you so much for the PR.
Can you please update the Description?
Also, I can review the PR once the tests pass.
-Thanks
Manthan

[josemakara2]

Close button accidentally clicked :)
Thanks @thesmallstar .
It is work in progress - will let you know when I consider the task complete.
I see my build failing as a result of format violations to changed files. I have configured eclipse fineratcdevprojectformatter but checking to fix format.

UPDATE: format fixed = no format violations.
Checked travis server and showing below unrelated fail message to this PR.

GlobalConfigurationTest > testGlobalConfigurations() FAILED
    java.lang.AssertionError: 1 expectation failed.
    Expected status code <200> but was <500>.
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Task :integration-tests:test FAILED
WARNING: The web application [fineract-provider] appears to have started a thread named [fineract_default_pool housekeeper] but has failed to stop it. This is very likely to create a memory leak. 
WARNING: The web application [fineract-provider] appears to have started a thread named [SimpleAsyncTaskExecutor-1407] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> Task :integration-tests:cargoStopLocal
FAILURE: Build failed with an exception.

Will carry on with the changes for now

[thesmallstar]

Looks like the integration test: testGlobalConfigurations is failing.

[thesmallstar]

If you think the tests that failed are not related to your PR (which happens quite a few times here :P ) close and reopen this PR to re-run tests. I will shortly link an issue here where you can report the failing test.

[thesmallstar]

Thanks for the update, tests pass now. Will review this tomorrow.

[josemakara2]

Thanks for the update, tests pass now. Will review this tomorrow.

No problems @thesmallstar, appreciated if you can please have a look as the work progresses. Many thanks!
I still believe this is work in progress and would not be a quick win to exhaustively get rid of SQL Injection vulnerabilities.
One way is to search the codebase on IDE for concatenations and fix to use parameterized queries.

The plan here is to use OWASP ZAP to automatically detect SQL Injections and attend to the list in the report from ZAP analysis.
FINERACT-969 has attached html report but that doesn't seem to have scanned Fineract APIs as the output is just on community-app.
I have excluded everything else except SQL Injection on ZAP to scan across Fineract API. I have had to quickly setup local test site with OAuth off to simplify things with OWASP ZAP.

A quick look on Alerts tab shows the detected violations. I will use the public link https://www.fineract.dev offered by @vorburger and produce the report which will be analysed for priority fixes in jira sub-tasks.

cc @vorburger

Some other things like below will possibly change the design to effectively parameterize the queries.

            sql.append("emo.error_message as errorMessage ");
            sql.append("from " + tableName() + " emo");

Ideally SQL string should only be built from string constants and every parameter inserted at runtime as bind variable (placeholder like ?, :name or @name) in the SQL string and its value set using the setX() methods of the PreparedStatement class.

Here tableName won't work as a parameter as FROM clause won't use setX() methods. This impacts both Security and Performance, and more performance hit if using databases that use a shared execution plan cache like PostgreSQL.

[josemakara2]

The first one to fix here will be as shown in the screenshot.
It was reported in 29 API URLs

> High	SQL Injection
Description	SQL injection may be possible.
URL	https://localhost:8443/fineract-provider/api/v1/users
Method	GET
Parameter	Fineract-Platform-TenantId
Attack	default%

The page results were successfully manipulated using the Boolean conditions [default%] and [%e%]

• Condition 1: Data was returned for the original parameter.
• Condition 2: No Data was returned for the manipulated parameter. %e%. Actually here it returned 2 rows whereas we expecting 1

SELECT  
  t.id, 
  t.timezone_id AS timezoneId , 
  t.name,
  t.identifier
FROM tenants t 
LEFT JOIN tenant_server_connections ts ON t.oltp_Id = ts.id  
WHERE t.identifier LIKE '%e%';

Followed by application error in tomcat logs ..
org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2 at org.springframework.dao.support.DataAccessUtils.nullableSingleResult(DataAccessUtils.java:100) at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:791) at org.apache.fineract.infrastructure.security.service.BasicAuthTenantDetailsServiceJdbc.loadTenantById(BasicAuthTenantDetailsServiceJdbc.java:143) at org.apache.fineract.infrastructure.security.filter.TenantAwareBasicAuthenticationFilter.doFilterInternal(TenantAwareBasicAuthenticationFilter.java:121) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

The vulnerability was detected by successfully restricting the data originally returned, by manipulating the parameter

Fix:
2 options

1. Add LIMIT 1 to the query
2. Replace like with = in the WHERE clause
   prefer no. 2 to be more perfomant

A explain on this query shows it is doing full-table scan on tenants.
Requires fix either

1. Index on column identifier or
2. Unique constraint on column identifier
   prefer no. ALTER TABLE tenants ADD UNIQUE (identifier);

[josemakara2]

@thesmallstar This is ready for review. I have updated jira FINERACT-969 related to OWASP ZAP tool.

There is additional fix (not related to SQL Injection) for duplicate entries error whilst running tests V365__reportCategoryList-FINERACT-1306.sql .
It was causing a 404 during as Tomcat couldn't not complete deployment after database error.

[thesmallstar]

@josemakara2 thanks for the PR, will review it ASAP.

[francisguchie]

@josemakara2 @thesmallstar
after a cherry-pick of this onto the latest develop branch, i have done a user test on reports and i do not get the SQL injection error mentioned in #1674 i give this a +1 on solving sql injection issues

[josemakara2]

@josemakara2 thanks for the PR, will review it ASAP.
Thanks @thesmallstar. Hope you had a nice Easter break. Checking with you when you are likely to finalise the review and provide feedback?
I understand most community members use US work timezone.
Thank you

[josemakara2]

@josemakara2 @thesmallstar
after a cherry-pick of this onto the latest develop branch, i have done a user test on reports and i do not get the SQL injection error mentioned in #1674 i give this a +1 on solving sql injection issues

And thanks for accepting to test as well.

1. It fixes the regression on reports - They are now working, and
2. Addresses SQL Injection issue

[thesmallstar]

LGTM!!
Does this PR cover all SQL string concatenations? I have removed a few last time...and I remember there to be a few more of this.

I see that you have not used the SQL builder module, It LGTM and the code is clean. I would still request @vorburger to see if that's okay.

This PR also covers https://issues.apache.org/jira/browse/FINERACT-1336 @francisguchie thanks for testing this!

[thesmallstar]

I have gone through the thread in detail, seems like this were some serious security issues floating around!
As you said in https://issues.apache.org/jira/browse/FINERACT-969 , this PR seems to cover a part of the 10 subtasks. It would be great if you could just edit the description with the specific tasks done..so that we can keep a track of it.

[thesmallstar]

Fix:
2 options

1. Add LIMIT 1 to the query
2. Replace like with = in the WHERE clause
   prefer no. 2 to be more perfomant

Yes here It makes sense to have =, I am not able to understand why was 'like' added in the first place, do you see any possible use-case we are breaking here?

A explain on this query shows it is doing full-table scan on tenants.
Requires fix either

1. Index on column identifier or
2. Unique constraint on column identifier
   prefer no. ALTER TABLE tenants ADD UNIQUE (identifier);

Wow this was a nice find!

The PR LGTM, I am requesting @vorburger and @ptuomola to also review this.. we can merge this ASAP :)

[josemakara2]

I have gone through the thread in detail, seems like this were some serious security issues floating around!
As you said in https://issues.apache.org/jira/browse/FINERACT-969 , this PR seems to cover a part of the 10 subtasks. It would be great if you could just edit the description with the specific tasks done..so that we can keep a track of it.

Thanks. I have edited pull request description to include specific tasks completed here

[josemakara2]

SQL string concatenations

Yes - it covers fixes to SQL string concatenations but not in the whole code base. That is why I am proposing to open sub-task in FINERACT-854 as explained in PR description.

[awasum]

LGTM. ping @vorburger , @ptuomola , @vidakovic . This looks interesting and important.

[awasum]

@josemakara2 Thank you very much for your fix. Keep up the good work you are doing at Apache Fineract. Make sure to join the mailing list. Your contributions are valuable. Welcome to the team.

[ptuomola]

Many thanks for contributing these changes - great work! Provided some feedback re the changes - would be great if you could take a look. Thanks!

[josemakara2]

Many thanks for contributing these changes - great work! Provided some feedback re the changes - would be great if you could take a look. Thanks!

@ptuomola
I have made the requested changes. Please have a look again.
Also see the separate PR in FINERACT-1345
Thank you.

[ptuomola]

Suggested change

	.append("(pcd.min_age <= GREATEST(datediff(?").append(",sch.duedate),0) and ").append("GREATEST(datediff(?")
	.append("(pcd.min_age <= GREATEST(datediff(?,sch.duedate),0) and GREATEST(datediff(?")

[josemakara2]

Thank you Petri.
😇
It wasn't in first review but if it improves the code base why not 🙂. Will add fix tonight when I get to my laptop with this project.

There are other things there which I didn't bother with initially but can improve readability. No change to functionality but helps understand things.
Prefer

• lowerCamel case for SQL functions, and uppercase for other keywords
• lowerCamel case for column names - dueDate not duedate
• Only break line if it violates 120 character line limit in the FINERACT eclipse formatter
• StringBuilder only if it is a conditional query, with if conditions, otherwise just a single query as below suffice.

Simply return

        public String schema() {
            return "" + 
                    "SELECT if(" + 
                    "    loan.loan_type_enum = 1, mclient.office_id, mgroup.office_id" + 
                    "  ) AS office_id," + 
                    "  loan.loan_type_enum," + 
                    "  pcd.criteria_id AS criteriaid," + 
                    "  loan.product_id," + 
                    "  loan.currency_code," + 
                    "  greatest(dateDiff(?, sch.duedate), 0) AS numberofdaysoverdue," + 
                    "  sch.duedate," + 
                    "  pcd.category_id," + 
                    "  pcd.provision_percentage," + 
                    "  loan.total_outstanding_derived AS outstandingbalance," + 
                    "  pcd.liability_account," + 
                    "  pcd.expense_account " + 
                    "FROM m_loan_repayment_schedule sch " + 
                    "LEFT JOIN m_loan loan ON sch.loan_id = loan.id " + 
                    "INNER JOIN m_loanproduct_provisioning_mapping lpm ON lpm.product_id = loan.product_id " + 
                    "INNER JOIN m_provisioning_criteria_definition pcd " + 
                    "  ON pcd.criteria_id = lpm.criteria_id " + 
                    "  AND (" + 
                    "    pcd.min_age <= greatest(dateDiff(?, sch.duedate), 0) AND greatest(dateDiff(?, sch.duedate), 0) <= pcd.max_age" + 
                    "  )" + 
                    "  AND pcd.criteria_id IS NOT NULL" + 
                    "LEFT JOIN m_client mclient ON mclient.id = loan.client_id " + 
                    "LEFT JOIN m_group mgroup ON mgroup.id = loan.group_id " + 
                    "WHERE loan.loan_status_id = 300 AND sch.duedate = (" + 
                    "  SELECT min(sch1.duedate) " + 
                    "  FROM m_loan_repayment_schedule sch1 " + 
                    "  WHERE sch1.loan_id=loan.id AND sch1.completed_derived = FALSE" + 
                    ")";
        }

But fineractdevprojectformatter won't allows this 😬 and instead its formats is not SQL readable.
as opposed to

            sqlQuery = new StringBuilder().append(
                    "select if(loan.loan_type_enum=1, mclient.office_id, mgroup.office_id) as office_id, loan.loan_type_enum, pcd.criteria_id as criteriaid, loan.product_id,loan.currency_code,")
                    .append("GREATEST(datediff(?")
                    .append(",sch.duedate),0) as numberofdaysoverdue,sch.duedate, pcd.category_id, pcd.provision_percentage,")
                    .append("loan.total_outstanding_derived as outstandingbalance, pcd.liability_account, pcd.expense_account from m_loan_repayment_schedule sch")
                    .append(" LEFT JOIN m_loan loan on sch.loan_id = loan.id")
                    .append(" JOIN m_loanproduct_provisioning_mapping lpm on lpm.product_id = loan.product_id")
                    .append(" JOIN m_provisioning_criteria_definition pcd on pcd.criteria_id = lpm.criteria_id and ")
                    .append("(pcd.min_age <= GREATEST(datediff(?,sch.duedate),0) and GREATEST(datediff(?")
                    .append(",sch.duedate),0) <= pcd.max_age) and ").append("pcd.criteria_id is not null ")
                    .append("LEFT JOIN m_client mclient ON mclient.id = loan.client_id ")
                    .append("LEFT JOIN m_group mgroup ON mgroup.id = loan.group_id ")
                    .append("where loan.loan_status_id=300 and sch.duedate = ")
                    .append("(select MIN(sch1.duedate) from m_loan_repayment_schedule sch1 where sch1.loan_id=loan.id and sch1.completed_derived=false)");

We can open a separate jira vote to fix formatter?

[thesmallstar]

thanks again @josemakara2! There was a lot of work behind this..

[BLasan]

@josemakara2 Would you mind telling the mysql version you've used?

[josemakara2]

@josemakara2 Would you mind telling the mysql version you've used?

@BLasan MySQL 8.0.23.
We have also tested this in MySQL 8.0.25 as it fixes some other unrelated issues we have been having with MySQL 5.7

[BLasan]

@josemakara2 Would you mind telling the mysql version you've used?

@BLasan MySQL 8.0.23.
We have also tested this in MySQL 8.0.25 as it fixes some other unrelated issues we have been having with MySQL 5.7

So the best option is to use mysql 8.0.25 right?

[BLasan]

@josemakara2 Would you mind telling the mysql version you've used?

@BLasan MySQL 8.0.23.
We have also tested this in MySQL 8.0.25 as it fixes some other unrelated issues we have been having with MySQL 5.7

I'm using java 14. Hope it's fine with this mysql version. I'm getting an error

* What went wrong:
Execution failed for task ':fineract-provider:createDB'.
> java.sql.SQLException: Could not connect: Client does not support authentication protocol requested by server; consider upgrading MySQL client

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

[ptuomola]

I don’t think we have tested Fineract with MySQL 8. At least when I did the MySQL upgrade last time, I tested only to 5.7. Which of course does not mean that it won’t work... But of course testing with MySQL 8 (as well as any patches required) would be a great idea - so please do go ahead Regards Petri

…

On 14 May 2021, at 3:57 PM, Benura Abeywardena ***@***.***> wrote: @josemakara2 <https://github.com/josemakara2> Would you mind telling the mysql version you've used? @BLasan <https://github.com/BLasan> MySQL 8.0.23. We have also tested this in MySQL 8.0.25 as it fixes some other unrelated issues we have been having with MySQL 5.7 I'm using java 14. Hope it's fine with this mysql version. I'm getting an error * What went wrong: Execution failed for task ':fineract-provider:createDB'. > java.sql.SQLException: Could not connect: Client does not support authentication protocol requested by server; consider upgrading MySQL client * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1671 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASJVCQYRH2NMS2ONA7USZ3TNTJVXANCNFSM4ZRMCVNA>.

[BLasan]

I don’t think we have tested Fineract with MySQL 8. At least when I did the MySQL upgrade last time, I tested only to 5.7. Which of course does not mean that it won’t work... But of course testing with MySQL 8 (as well as any patches required) would be a great idea - so please do go ahead Regards Petri
…
On 14 May 2021, at 3:57 PM, Benura Abeywardena @.***> wrote: @josemakara2 https://github.com/josemakara2 Would you mind telling the mysql version you've used? @BLasan https://github.com/BLasan MySQL 8.0.23. We have also tested this in MySQL 8.0.25 as it fixes some other unrelated issues we have been having with MySQL 5.7 I'm using java 14. Hope it's fine with this mysql version. I'm getting an error * What went wrong: Execution failed for task ':fineract-provider:createDB'. > java.sql.SQLException: Could not connect: Client does not support authentication protocol requested by server; consider upgrading MySQL client * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1671 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASJVCQYRH2NMS2ONA7USZ3TNTJVXANCNFSM4ZRMCVNA.

Tried mysql 5.7 as well. And the gradle version has been downgraded in the latest codebase to 6.8. It's not using gradle 7.0 which was perfectly executed without any issue. After rebasing I got this error

[BLasan]

@ptuomola Another question is why we have downgraded the gradle version?

[ptuomola]

I don’t think anyone has downgraded Gradle. The latest version that Fineract has been tested with was 6.8.3. There is an automatic pull request from Renovate to upgrade to Gradle 7.0 and at least that has failed - which would suggest some changes are required: #1690 <#1690>. But I haven’t looked at the reasons for failure - perhaps it’s an easy fix? Please feel free to do testing with Gradle 7.0 and submit a pull request with any changes required for it to work... Regards Petri

…

On 14 May 2021, at 4:05 PM, Benura Abeywardena ***@***.*** ***@***.***>> wrote: @ptuomola <https://github.com/ptuomola> Another question is why we have downgraded the gradle version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1671 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASJVCWW5EFBX3AOY5HNJWDTNTKUFANCNFSM4ZRMCVNA>.

[josemakara2]

@BLasan
I see Travis CI run MySQL 5.7 when it was merged. So this should work with MySQL 5.7
https://travis-ci.com/github/apache/fineract/builds/222875519

This link has 3 options for the issue you are facing and might help you run Fineract in MySQL 5.7.x series
https://dev.mysql.com/doc/refman/5.6/en/old-client.html
https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-versions.html