slow export

[tsah]

Export is very slow when dealing with a big number of entries. It took me about 6 minutes to export 4000 entries. Is there any way to speed it up?

[bbenezech]

Export needs a few things:

• auto-include association (you may have n+1 issues)
• batch treatment (loading 4k objects in memory is probably a bad idea)
• AJAX processing (even if user waits, front servers will timeout anyway)

Any PR welcome.

[outworlder]

Indeed, I've just run into this problem. Heroku will terminate requests after 30 seconds.

[rmcastil]

I'm currently running into this problem and would be up for creating a PR for fixing it.

@bbenezech does the rails_admin team currently have a plan on how they wanted to solve this issue?

[tensiondriven]

@sferik - Any ideas on how to work around this timeout issue on Heroku? Either by optimizing queries, adding indexies, or (worst case) forking rails_admin to use something like sidekiq to generate requested CSV's asynchronously and send them by email? This is a major issue for companies using rails_admin to power their operations/customer service. (Using Mongoid fwiw)

[tensiondriven]

Could this be best and most easily fixed by changing the export code to use streaming?

From https://devcenter.heroku.com/articles/request-timeout : "Cedar supports HTTP 1.1 features such as long-polling and streaming responses. An application has an initial 30 second window to respond with a single byte back to the client. However, each byte transmitted thereafter (either received from the client or sent by your application) resets a rolling 55 second window. If no data is sent during the 55 second window, the connection will be terminated."

[sferik]

Request streaming is only supported by Rails 4. I'm not quite ready to remove Rails 3 support for this feature but feel free to put it in a branch. As for doing the work in the background, that adds a lot of complexity.

RailsAdmin isn't intended to be your app, it's intended to be an admin interface for your app. It was never designed to "power [your] operations/customer service". You might want to find or build another tool to handle your exports if that's essential to the business.

[natarius]

@sferik I assume all the export code is in https://github.com/sferik/rails_admin/blob/master/lib/rails_admin/config/actions/export.rb

right!?

[natarius]

I would try to build a solution that creates the file in a sidekiq task (if sidekiq is available) and the browser would just poll the server every X seconds until its ready.

Opinions?

[marks]

+1 .. has anyone implemented a streaming or background solution before I go ahead and try to tackle this?

[natarius]

Nope....didn't come up with a simple enough design to not add too many dependencies (sidekiq/resque, S3, etc)

But would still love to have a solution for this :)

[mshibuya]

#1783 might partially solve this problem.

[dalpo]

#1830

[robertjwhitney]

Interested in helping with eager loading. Right now rails admin only eager loads belongs_to on a collection by design (hard coded). Thoughts on the design for making this optional, maybe specifically for export?

 def get_collection(model_config, scope, pagination)
      associations = model_config.list.fields.select {|f| f.type == :belongs_to_association && !f.polymorphic? }.map {|f| f.association[:name] }
      options = {}
      options = options.merge(:page => (params[Kaminari.config.param_name] || 1).to_i, :per => (params[:per] || model_config.list.items_per_page)) if pagination
      options = options.merge(:include => associations) unless associations.blank?
      options = options.merge(get_sort_hash(model_config))
      options = options.merge(:query => params[:query]) if params[:query].present?
      options = options.merge(:filters => params[:f]) if params[:f].present?
      options = options.merge(:bulk_ids => params[:bulk_ids]) if params[:bulk_ids]

      objects = model_config.abstract_model.all(options, scope)
    end

[robertjwhitney]

Adding some sort of option to list_entries, maybe?

[grillermo]

I placed a 100 dollars bounty to anybody that implements ActiveJob for this, with email after finishing the export or possibly a rails admin action where you get a link to download the csv, json, from an S3 server.

[PapePathe]

@grillermo you mean rails 4 ActiveJob?

[grillermo]

@PapePathe yes, i just edited the message with that correction.

[grillermo]

@tsah Would you change the issue title to help the bounty please? i suggest Implement background job for exports in rails admin, or just adding the word [bounty] would help a lot, thank you.

[babasbot]

Maybe is not the best way to go, but what about using a child process? go-tandem#2

[mshibuya]

Associations other than belongs_to can be eager-loaded via configuration now, use

field :players do
  eager_load true
end

[ntloi95]

@mshibuya how to use your code? Where to put it? In the model, rights?

field :players do
  eager_load true
end

[timomeh]

In case someone stumbles across this and wants to export huge amounts of records: You can write your own export with batches and a streaming response, using a custom action.

1. You need to add ActionController::Live to rails_admin's controller:

# config/initializers/rails_admin.rb

Dir[Rails.root.join('app', 'rails_admin', '**/*.rb')].each { |file| require file }

module RailsAdmin
  class EnhancedController < ActionController::Base
    include ActionController::Live
  end
end

RailsAdmin.config do |config|
  # ...

  config.actions do
    # ...
    custom_export
  end

  config.parent_controller = '::RailsAdmin::EnhancedController'

2. Create a custom action. Use RailsAdmin::Config::Actions::Base as a guide. For example:

# app/rails_admin/config/actions/custom_export.rb

module RailsAdmin
  module Config
    module Actions
      class CustomExport < RailsAdmin::Config::Actions::Base
        RailsAdmin::Config::Actions.register(self)

        register_instance_option :link_icon do
          'icon-share'
        end

        register_instance_option :collection? do
          true # Makes action tab visible for the collection
        end

        register_instance_option :http_methods do
          %i[get post]
        end

        register_instance_option :controller do
          proc do
            if request.get?
              render action: @action.template_name

            elsif request.post?
              from = Time.zone.parse(params[:from])

              # Prepare headers for streaming
              response.headers.delete('Content-Length')
              response.headers['Cache-Control'] = 'no-cache'
              response.headers['X-Accel-Buffering'] = 'no'
              response.headers['Content-Type'] = 'text/event-stream'

              # There's an issue in rack where ActionController::Live doesn't work with the ETags middleware
              # See https://github.com/rack/rack/issues/1619#issuecomment-606315714
              response.headers['ETag'] = '0'
              response.headers['Last-Modified'] = '0'

              # Download response stream into a file
              response.headers['Content-Disposition'] = "attachment; filename=testfile.txt"

              SomeModel.where('created_at > ?', from).find_each(batch_size: 500) do |record|
                # Define how the records should be exported
                response.stream.write record.to_s
              end
            ensure
              response.stream.close
            end
          end
        end
      end
    end
  end
end

3. Create a corresponding view. Use rails_admin's views as a guide. For example:

-# app/views/rails_admin/main/custom_export.html.haml
 
= form_tag custom_export_path, class: 'form-horizontal' do
  .form-group.control-group
    %label.col-sm-2.control-label{for: "from"}= t('admin.custom_export.from')
    .col-sm-10.controls
      = text_field_tag :from, Time.now.beginning_of_day - 1.month, required: true, data: { datetimepicker: true, options: { "showTodayButton":true,"format":"YYYY-MM-DD HH:mm" } }, class: 'form-control'

  .form-group.form-actions
    .col-sm-offset-2.col-sm-10
      %button.btn.btn-primary{type: "submit"}
        %i.icon-white.icon-download
        = t("admin.custom_export.submit")

That should do it. Maybe it saves someones time and nerves.

[grillermo]

thanks @timomeh that's awesome

[coorasse]

I believe streaming the response should be default on rails_admin