Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload Service 4 #450

Closed
30 of 38 tasks
gotev opened this issue Aug 14, 2019 · 4 comments
Closed
30 of 38 tasks

Upload Service 4 #450

gotev opened this issue Aug 14, 2019 · 4 comments
Assignees
Labels
ongoing Issue which is in progress task Something which needs to be done. Not necessarily a bug or an enhancement
Milestone

Comments

@gotev
Copy link
Owner

gotev commented Aug 14, 2019

  • Android 10 Support. Set Target API to 29 https://developer.android.com/about/dashboards
  • Upgrade all the dependencies to their latest versions for maximum stability and support.
    • This Includes OkHttp 4.x which Min SDK is 21. This poses some limitations for old devices support, but the world needs to move forward, mainly for security and maintenance reasons. API < 21 devices have many problems supporting latest TLS ciphers. Many of them cannot at all. This is an upload library, safety comes first. My oldest hardware device has Android 5.0 (2014) and my apps are all API 21+. No contributor or supporter for older devices than 5.0 to date, so I cannot afford providing support of any kind for those versions myself. If someone is willing to help, let me know.
  • Migrate the codebase to Kotlin to ease maintenance, evolution and provide null safety
  • Code refactoring. The codebase has had many tiny modifications going forward, but looking at the whole picture now, there's a call for change. Google came up with many innovations and some JetPack components may be well suited for this library.
  • Use latest Gradle 5.x together with new gradle maven plugin from https://github.com/sky-uk/gradle-maven-plugin
  • Public API changes: delegates as they are implemented (3.5.2) have been a major source of issues and misdirection (even if there's a Wiki for that), as they don't support device rotation and user navigating away from the activity. They will be dropped and replaced by something better and less misleading.
  • Design and implement delegates substitutes
  • As explained in Add ability to customize notification channel name & group #382 there's a design mistake in how notification channels are handled. From 4.0.0 upload service will only use already created notification channels and throw errors if no channel is provided or if the provided channel ID does not exist.
  • Add possibility to register custom scheme handlers to handle other schemes which are not file or content via UploadServiceConfig.addSchemeHandler("yourscheme", YourHandler::class.java)
  • Move all the global configurations in the separated UploadServiceConfig class
  • Make default retries for each request configurable in UploadServiceConfig
  • Enable UTF8 Charset by default (up to 3.5.2 you had to manually enable it)
  • Fix cancelling upload tasks from notification on Android API 26+
  • Change the task events lifecycle to be more Rx and LiveData-like, with the following events
    • onProgress
    • onSuccess or onError
    • onCompleted (which comes always after onSuccess or onError)
    • task cancellation -> UserCancelledUploadException passed in onError
    • error response from server -> UploadError(serverResponse) passed in onError
  • Set Log level to DEBUG or OFF depending on BuildConfig variable value with UploadServiceLogger.setDevelopmentMode(BuildConfig.DEBUG)
  • Rename Broadcast Receiver to Request Observer
  • move uploadId parameter out of request builder constructors. Custom upload IDs can be set by calling .setUploadId("yourId") on request builder
  • Add method setBearerAuth("bearerToken") to HTTP upload builders (multipart, binary)
  • Fix NPEs in FTP Upload Module
  • Use LocalBroadcastManager instead of global to notify upload task events. This reduces overhead and augments security
  • Make ThreadPool overridable to allow library users to use shared ThreadPool
  • Custom and localizabile placeholders (Placeholders localization #466)
  • Custom notification handler. It's possible to use your own notification handler implementation.
  • Make it easier to add and handle notification actions, by supplying a default broadcast receiver which gets broadcast pending intents and the uploadId when building notification configuration
  • Remove setCustomUserAgent from request builders and move the notion of User-Agent to the Http Stack implementations
  • Other things which may pop up while doing the above

Docs, debugging and issues

  • Log exception which caused an upload retry
  • Provide step by step tutorials to migrate to 4.x
  • Improve debug logs to ease troubleshooting
  • New documentation and examples in Kotlin
  • Make a JSON representation for each of the bundled upload requests (multipart, binary, ftp)
  • New issue templates, for both 3.x and 4.x versions

Demo App

  • Upgrade to latest Recycler Adapter
  • Add example usage for system file picker with content resolver
  • Use Kotlin Extensions and remove ButterKnife
  • Allow choosing between File Picker and Content Provider Picker
  • Allow choosing the HTTP Stack (Hurl or OkHttp)
@gotev gotev added the task Something which needs to be done. Not necessarily a bug or an enhancement label Aug 14, 2019
@gotev gotev self-assigned this Aug 14, 2019
@gotev gotev added this to the 4.0.0 milestone Aug 14, 2019
gotev added a commit that referenced this issue Aug 15, 2019
…laced dcendents maven plugin with new gradle-maven-plugin from sky-uk. Added kotlin support in the modules and bumped versions for new development cycles. Added script for local snapshot release. Updated RecyclerAdapter.
gotev added a commit that referenced this issue Aug 15, 2019
…safely handle it. Refactored createBody method.
@gotev gotev added the ongoing Issue which is in progress label Aug 15, 2019
gotev added a commit that referenced this issue Aug 16, 2019
gotev added a commit that referenced this issue Aug 16, 2019
…otlin and moved to network package. Setup kotlin in demo app and added global broadcast receiver.
gotev added a commit that referenced this issue Aug 17, 2019
gotev added a commit that referenced this issue Aug 17, 2019
…r custom scheme handlers. Improved exception messages. Started migration of parcelable data classes to Kotlin.
gotev added a commit that referenced this issue Aug 17, 2019
…hanced debug log messages by printing upload service configuration, android version and processor cores
gotev added a commit that referenced this issue Aug 18, 2019
…e handlers names. Dropped ContentType class in favor of a String extension.
gotev added a commit that referenced this issue Aug 18, 2019
gotev added a commit that referenced this issue Aug 18, 2019
gotev added a commit that referenced this issue Aug 18, 2019
…delegate debug log to info because some devices (e.g. Huawei ALE-L21 with Android API 22) does not display Log.d in LogCat and some vendor specific settings are needed. It's better to not deal with those.
gotev added a commit that referenced this issue Aug 21, 2019
gotev added a commit that referenced this issue Sep 4, 2019
gotev added a commit that referenced this issue Sep 28, 2019
gotev added a commit that referenced this issue Sep 28, 2019
…static methods java friendly. Migrated HttpUploadTaskParameters to Kotlin. Added extensions for multipart upload files.
@gotev
Copy link
Owner Author

gotev commented Sep 28, 2019

After a month of hard work on this migration, 4.0.0 alpha 1 is almost done!

Kotlin codebase looks a lot more polished and readable than it was in Java. New features are available and there's room for more. Many massive refactorings and a lot of testing to preserve public APIs as much as possibile and to port existing functionalities without regressions.

Codebase is tiny now, more maintainable and modular. I've used cloc to have some rough data about lines of code saved.

UploadService Core package

Before

github.com/AlDanial/cloc v 1.82  T=0.11 s (341.3 files/s, 43423.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                            36            760           1185           2635
-------------------------------------------------------------------------------
SUM:                            36            760           1185           2635
-------------------------------------------------------------------------------

After

github.com/AlDanial/cloc v 1.82  T=0.07 s (684.8 files/s, 43468.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Kotlin                          46            458            710           1752
-------------------------------------------------------------------------------
SUM:                            46            458            710           1752
-------------------------------------------------------------------------------

UploadService OkHttp package

Before

github.com/AlDanial/cloc v 1.82  T=0.02 s (160.1 files/s, 11205.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                             3             45             12            153
-------------------------------------------------------------------------------
SUM:                             3             45             12            153
-------------------------------------------------------------------------------

After

github.com/AlDanial/cloc v 1.82  T=0.02 s (232.4 files/s, 9412.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Kotlin                           4             32             17            113
-------------------------------------------------------------------------------
SUM:                             4             32             17            113
-------------------------------------------------------------------------------

UploadService FTP package

Before

github.com/AlDanial/cloc v 1.82  T=0.03 s (140.9 files/s, 30362.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                             4            148            213            501
-------------------------------------------------------------------------------
SUM:                             4            148            213            501
-------------------------------------------------------------------------------

After

github.com/AlDanial/cloc v 1.82  T=0.03 s (220.4 files/s, 26113.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Kotlin                           6            106            153            452
-------------------------------------------------------------------------------
SUM:                             6            106            153            452
-------------------------------------------------------------------------------

Overall lines of code

Before: 3209
After: 2317

Almost 28% less code!

Thanks to https://android.jlelse.eu/writing-java-friendly-kotlin-code-c408b24fb4e for Kotlin Java Friendly code advises.

@gotev gotev mentioned this issue Sep 28, 2019
gotev added a commit that referenced this issue Sep 28, 2019
gotev added a commit that referenced this issue Sep 29, 2019
…oved Apache Commons FTP Code in FTPClientWrapper. Ported FTPUploadTask to Kotlin. Newer macOS doesn't have FTP server anymore, so setup a simple script to use vsftpd inside docker.
gotev added a commit that referenced this issue Sep 29, 2019
* #450 upgraded gradle plugin to 3.4.2, gradle wrapper to 5.4.1 and replaced dcendents maven plugin with new gradle-maven-plugin from sky-uk. Added kotlin support in the modules and bumped versions for new development cycles. Added script for local snapshot release. Updated RecyclerAdapter.

* update travis configuration

* update travis configuration

* #450 ported OkHttp module to Kotlin and upgraded to OkHttp 4.0.1

* #450 OkHttp BodyWriter is now cloaseable. Relying on Kotlin "use" to safely handle it. Refactored createBody method.

* #450 migrated HurlStack and generic http facade to Kotlin

* #450 try to make travis happy again

* #450 Refactored http package to network. Migrated ServerResponse to Kotlin and moved to network package. Setup kotlin in demo app and added global broadcast receiver.

* #450 migrated logger to Kotlin

* #450 better name for request body writer delegate method

* #450 migrated Scheme Handlers to Kotlin. It's now possible to register custom scheme handlers. Improved exception messages. Started migration of parcelable data classes to Kotlin.

* #450 moved all the configurable parameters in UploadServiceConfig. Enhanced debug log messages by printing upload service configuration, android version and processor cores

* #450 migrated UploadFile and Placeholders to Kotlin. Refactored scheme handlers names. Dropped ContentType class in favor of a String extension.

* #450 do not parcelize utility property bodyString

* #450 migrated BroadcastData in Kotlin

* #450 migrated UploadInfo to Kotlin.

* #450 Moved SchemeHandlers to UploadServiceConfig. Set default logger delegate debug log to info because some devices (e.g. Huawei ALE-L21 with Android API 22) does not display Log.d in LogCat and some vendor specific settings are needed. It's better to not deal with those.

* #450 Migrated UploadNotificationAction to Kotlin

* #450 fix cancelling uploads on Android API 26+

* #450 migrated UploadTaskParameters to Kotlin and refactored where needed

* #450 updated gradle plugin

* #450 updated Kotlin to 1.3.50. Used require to make code more readable.

* #450 migrated binary upload task to kotlin. Added http URL validator extension for strings

* #450 moved notification handling in separate listener. Removed automatic notification channel creation on Android 8+. This reduces UploadTask competences, complexity and code.

* #450 renamed package notifier to tasklistener

* #450 moved broadcast operations out of UploadTask. This further simplifies UploadTask logic and competences.

* #450 file deletion is now a competence of the scheme handler. This also solves a potential bug in FTP upload module and allows file deletion via content resolver (if supported). UploadInfo now contains full UploadFile information instead of the string path.

* #450 remove static constants in UploadTask in favor of a ServerResponse factory method

* #450 upload task now receives the notification ID as an init parameter

* #450 moved throttling logic in separate function

* #450 converted UploadTask in Kotlin

* #450 default number of retries for each request is now configurable in Retry Policy global configuration. Dropped Upload Delegate implementation.

* #450 Removed example empty UploadReceiver. BroadcastReceivers are now called RequestObservers. Log level can now be tuned based on BuildConfig.DEBUG value. Removed OkHttp call timeout. Not suitable for uploads which can run an indefinite amount of time. Started implementation of new kind of broadcast receivers. Added utility inline fun to safely perform the same action on all of the task's observers.

* #450 create new uploadInfo for each observer

* #450 http uploads now uses UTF-8 by default. removed concept of Intent from UploadTasks. Additional params are now passed directly in UploadTaskParameters. Converted UploadService to Kotlin

* #450 refactor starting upload. Reduced code in UploadService.

* #450 better log messages when instantiating new task fails

* #450 task observers are now passed as parameters in the init method. This in turn allowed to make the task independent from the upload service.

* #450 code shrinking

* #450 simplified notifying progress by sending only the bytes transferred and let the base class handle the rest

* #450 inject http stack in upload tasks. Done some renaming of http stack data types and methods to make the code more fluent to read

* #450 calculate sleep deadline only once

* #450 make HttpRequest implement the Closeable interface

* #450 use better equals

* #450 using lambda functions in logger to save performance evaluating messages which are not going to be logged. Started conversion of HttpUploadTask to Kotlin.

* #450 http stack requests auto-close on success and on error

* #450 converted MultipartUploadTask to Kotlin

* #450 code reorder

* #450 remove unnecessary annotation in Kotlin

* #450 progress listener is now passed to the BodyWriter which manages it internally. Simplified multipart upload task. Simplified notification handler.

* #450 Changed the task events lifecycle to be more Rx and LiveData-like

* #450 implemented tagging each HTTP connection with the uploadID to have better logging

* #450 refactored successfully uploaded files logic. Moved wake lock logic in extensions

* #450 removed author comments as they can be read from github, as well as some old docs

* #450 refactor sending broadcast

* #40 ported UploadNotificationConfig and UploadNotificationStatusConfig in Kotlin. Made necessary refactorings.

* #450 migrated all the request builders to Kotlin

* #450 made UploadServiceConfig, UploadServiceLogger and UploadService static methods java friendly. Migrated HttpUploadTaskParameters to Kotlin. Added extensions for multipart upload files.

* #450 migrated FTPUploadTaskParameters and UnixPermissions to Kotlin. Added Unit Tests for UnixPermissions.

* #450 moved content provider file selector in files picker activity. Moved Apache Commons FTP Code in FTPClientWrapper. Ported FTPUploadTask to Kotlin. Newer macOS doesn't have FTP server anymore, so setup a simple script to use vsftpd inside docker.
gotev added a commit that referenced this issue Sep 29, 2019
gotev added a commit that referenced this issue Oct 20, 2019
…oadcastReceiver) Lifecycle aware and adding addional onCompletedWhileNotObserving method to cover an edge case
gotev added a commit that referenced this issue Oct 20, 2019
gotev added a commit that referenced this issue Oct 20, 2019
… task event. This reduces overhead and augments security.
This was referenced Nov 10, 2019
@gotev
Copy link
Owner Author

gotev commented Jan 15, 2020

Closing this. Remaining open points will be address incrementally with new versions

@lock
Copy link

lock bot commented Jun 24, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related issues, referencing this one.

1 similar comment
@lock
Copy link

lock bot commented Jun 24, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related issues, referencing this one.

@lock lock bot locked as resolved and limited conversation to collaborators Jun 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
ongoing Issue which is in progress task Something which needs to be done. Not necessarily a bug or an enhancement
Projects
None yet
Development

No branches or pull requests

1 participant