[Feature][ActiveStorage] add option for proxying files to allow CDN caching

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Feature][ActiveStorage] add option for proxying files to allow CDN caching

Jonathan Fleckenstein
ActiveStorage should give users the option to proxy files. This would allow files to be cached by a CDN (image load times for my application went from 500+ms to 50ms) 🚀. Something along the lines of this is what I'm proposing:

# frozen_string_literal: true
module ActiveStorage

  class BlobsController < BaseController
    include ActiveStorage::SetBlob

    def show
      if ActiveStorage.redirect_to_service_url
        expires_in ActiveStorage.service_urls_expire_in
        redirect_to @blob.service_url(disposition: params[:disposition])
      else
        stream_blob
      end
    end

    def stream_blob
      expires_in ActiveStorage.service_urls_expire_in, public: true
      response.headers['Content-Type'] = params[:content_type]
      response.headers['Content-Disposition'] = params[:disposition]

      @blob.download do |chunk|
        response.stream.write(chunk)
      end
    ensure
      response.stream.close
    end
  end
end


Users looking for image loading with a CDN could set this up by setting

app.config.active_storage.redirect_to_service_url = false
app.config.active_storage.service_urls_expire_in = 1.year

I just threw this together for a project I'm working on, but I would be happy to clean this up and make a proper pull request if the Rails team is open to the idea. There's a fair amount of people requesting this feature https://github.com/rails/rails/issues/31419 some of the solutions in this issue are directly linking to the service files, but streaming would keep the service provider abstracted away while also allowing for proper CDN caching.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature][ActiveStorage] add option for proxying files to allow CDN caching

George Claghorn-2
+1 for supporting publicly-available files. We’re almost certainly going to do that at some point.

We needn’t tie it to proxying downloads, though. For larger apps, it’s no more desirable to take on the substantial cost of proxying for public files any more than for private ones. You can put a CDN in front of a public S3 or GCS bucket.

On Fri, Nov 2, 2018 at 10:17 PM Jonathan Fleckenstein <[hidden email]> wrote:
ActiveStorage should give users the option to proxy files. This would allow files to be cached by a CDN (image load times for my application went from 500+ms to 50ms) 🚀. Something along the lines of this is what I'm proposing:

# frozen_string_literal: true
module ActiveStorage

  class BlobsController < BaseController
    include ActiveStorage::SetBlob

    def show
      if ActiveStorage.redirect_to_service_url
        expires_in ActiveStorage.service_urls_expire_in
        redirect_to @blob.service_url(disposition: params[:disposition])
      else
        stream_blob
      end
    end

    def stream_blob
      expires_in ActiveStorage.service_urls_expire_in, public: true
      response.headers['Content-Type'] = params[:content_type]
      response.headers['Content-Disposition'] = params[:disposition]

      @blob.download do |chunk|
        response.stream.write(chunk)
      end
    ensure
      response.stream.close
    end
  end
end


Users looking for image loading with a CDN could set this up by setting

app.config.active_storage.redirect_to_service_url = false
app.config.active_storage.service_urls_expire_in = 1.year

I just threw this together for a project I'm working on, but I would be happy to clean this up and make a proper pull request if the Rails team is open to the idea. There's a fair amount of people requesting this feature https://github.com/rails/rails/issues/31419 some of the solutions in this issue are directly linking to the service files, but streaming would keep the service provider abstracted away while also allowing for proper CDN caching.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature][ActiveStorage] add option for proxying files to allow CDN caching

Jonathan Fleckenstein
The large cost really depends on your cache hit rate, for my personal apps and my employer (a high traffic e-commerce site) static assets have low miss rate as they never change and have long expiring headers. How about giving the user the option, while keeping redirect as the default?

ActiveStorage.delivery_method = :redirect # with options to set it to :proxy or :direct

redirect: 
  • default
  • semi private
  • keeps provider abstraction
  • slow image loading
  • not CDN friendly
  • keep existing service_urls_expire_in
proxy:
  • CDN friendly
    • no configuration when used with cloudflare and other similar CDNs
  • keeps provider abstraction
  • could increase application server load
  • defaults to a long service_urls_expire_in
direct:
  • CDN friendly
    • it takes more configuration than the proxy method
  • breaks provider abstraction
    • Changing providers will require a fair amount of config changes to work with CDN
  • less load on application server
  • defaults to a long service_urls_expire_in
On Friday, November 2, 2018 at 10:36:58 PM UTC-4, George Claghorn wrote:
+1 for supporting publicly-available files. We’re almost certainly going to do that at some point.

We needn’t tie it to proxying downloads, though. For larger apps, it’s no more desirable to take on the substantial cost of proxying for public files any more than for private ones. You can put a CDN in front of a public S3 or GCS bucket.

On Fri, Nov 2, 2018 at 10:17 PM Jonathan Fleckenstein <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="rxz9FC2SCQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jonfl...@...> wrote:
ActiveStorage should give users the option to proxy files. This would allow files to be cached by a CDN (image load times for my application went from 500+ms to 50ms) 🚀. Something along the lines of this is what I'm proposing:

# frozen_string_literal: true
module ActiveStorage

  class BlobsController < BaseController
    include ActiveStorage::SetBlob

    def show
      if ActiveStorage.redirect_to_service_url
        expires_in ActiveStorage.service_urls_expire_in
        redirect_to @blob.service_url(disposition: params[:disposition])
      else
        stream_blob
      end
    end

    def stream_blob
      expires_in ActiveStorage.service_urls_expire_in, public: true
      response.headers['Content-Type'] = params[:content_type]
      response.headers['Content-Disposition'] = params[:disposition]

      @blob.download do |chunk|
        response.stream.write(chunk)
      end
    ensure
      response.stream.close
    end
  end
end


Users looking for image loading with a CDN could set this up by setting

app.config.active_storage.redirect_to_service_url = false
app.config.active_storage.service_urls_expire_in = 1.year

I just threw this together for a project I'm working on, but I would be happy to clean this up and make a proper pull request if the Rails team is open to the idea. There's a fair amount of people requesting this feature <a href="https://github.com/rails/rails/issues/31419" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fissues%2F31419\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNElnbgw_As5Yrk4o9ggJ92U8mRzRQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fissues%2F31419\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNElnbgw_As5Yrk4o9ggJ92U8mRzRQ&#39;;return true;">https://github.com/rails/rails/issues/31419 some of the solutions in this issue are directly linking to the service files, but streaming would keep the service provider abstracted away while also allowing for proper CDN caching.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="rxz9FC2SCQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonrails-co...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="rxz9FC2SCQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonra...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/rubyonrails-core" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;">https://groups.google.com/group/rubyonrails-core.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature][ActiveStorage] add option for proxying files to allow CDN caching

George Claghorn-2
I’m okay with adding proxy support in general, but a global setting seems less than ideal (unless it’s paired with a way to override it on a per-file basis).

On Fri, Nov 2, 2018 at 11:22 PM Jonathan Fleckenstein <[hidden email]> wrote:
The large cost really depends on your cache hit rate, for my personal apps and my employer (a high traffic e-commerce site) static assets have low miss rate as they never change and have long expiring headers. How about giving the user the option, while keeping redirect as the default?

ActiveStorage.delivery_method = :redirect # with options to set it to :proxy or :direct

redirect: 
  • default
  • semi private
  • keeps provider abstraction
  • slow image loading
  • not CDN friendly
  • keep existing service_urls_expire_in
proxy:
  • CDN friendly
    • no configuration when used with cloudflare and other similar CDNs
  • keeps provider abstraction
  • could increase application server load
  • defaults to a long service_urls_expire_in
direct:
  • CDN friendly
    • it takes more configuration than the proxy method
  • breaks provider abstraction
    • Changing providers will require a fair amount of config changes to work with CDN
  • less load on application server
  • defaults to a long service_urls_expire_in
On Friday, November 2, 2018 at 10:36:58 PM UTC-4, George Claghorn wrote:
+1 for supporting publicly-available files. We’re almost certainly going to do that at some point.

We needn’t tie it to proxying downloads, though. For larger apps, it’s no more desirable to take on the substantial cost of proxying for public files any more than for private ones. You can put a CDN in front of a public S3 or GCS bucket.

On Fri, Nov 2, 2018 at 10:17 PM Jonathan Fleckenstein <[hidden email]> wrote:
ActiveStorage should give users the option to proxy files. This would allow files to be cached by a CDN (image load times for my application went from 500+ms to 50ms) 🚀. Something along the lines of this is what I'm proposing:

# frozen_string_literal: true
module ActiveStorage

  class BlobsController < BaseController
    include ActiveStorage::SetBlob

    def show
      if ActiveStorage.redirect_to_service_url
        expires_in ActiveStorage.service_urls_expire_in
        redirect_to @blob.service_url(disposition: params[:disposition])
      else
        stream_blob
      end
    end

    def stream_blob
      expires_in ActiveStorage.service_urls_expire_in, public: true
      response.headers['Content-Type'] = params[:content_type]
      response.headers['Content-Disposition'] = params[:disposition]

      @blob.download do |chunk|
        response.stream.write(chunk)
      end
    ensure
      response.stream.close
    end
  end
end


Users looking for image loading with a CDN could set this up by setting

app.config.active_storage.redirect_to_service_url = false
app.config.active_storage.service_urls_expire_in = 1.year

I just threw this together for a project I'm working on, but I would be happy to clean this up and make a proper pull request if the Rails team is open to the idea. There's a fair amount of people requesting this feature https://github.com/rails/rails/issues/31419 some of the solutions in this issue are directly linking to the service files, but streaming would keep the service provider abstracted away while also allowing for proper CDN caching.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature][ActiveStorage] add option for proxying files to allow CDN caching

Jonathan Fleckenstein
Global with individual override would be perfect! I’ll try to put this together and make a PR in the next couple days/weeks. Btw, David has been grilling everybody on Twitter lately about working too much, might want to be careful replying to this mailing list on a Friday night....😀

On Fri, Nov 2, 2018 at 11:29 PM George Claghorn <[hidden email]> wrote:
I’m okay with adding proxy support in general, but a global setting seems less than ideal (unless it’s paired with a way to override it on a per-file basis).

On Fri, Nov 2, 2018 at 11:22 PM Jonathan Fleckenstein <[hidden email]> wrote:
The large cost really depends on your cache hit rate, for my personal apps and my employer (a high traffic e-commerce site) static assets have low miss rate as they never change and have long expiring headers. How about giving the user the option, while keeping redirect as the default?

ActiveStorage.delivery_method = :redirect # with options to set it to :proxy or :direct

redirect: 
  • default
  • semi private
  • keeps provider abstraction
  • slow image loading
  • not CDN friendly
  • keep existing service_urls_expire_in
proxy:
  • CDN friendly
    • no configuration when used with cloudflare and other similar CDNs
  • keeps provider abstraction
  • could increase application server load
  • defaults to a long service_urls_expire_in
direct:
  • CDN friendly
    • it takes more configuration than the proxy method
  • breaks provider abstraction
    • Changing providers will require a fair amount of config changes to work with CDN
  • less load on application server
  • defaults to a long service_urls_expire_in
On Friday, November 2, 2018 at 10:36:58 PM UTC-4, George Claghorn wrote:
+1 for supporting publicly-available files. We’re almost certainly going to do that at some point.

We needn’t tie it to proxying downloads, though. For larger apps, it’s no more desirable to take on the substantial cost of proxying for public files any more than for private ones. You can put a CDN in front of a public S3 or GCS bucket.

On Fri, Nov 2, 2018 at 10:17 PM Jonathan Fleckenstein <[hidden email]> wrote:
ActiveStorage should give users the option to proxy files. This would allow files to be cached by a CDN (image load times for my application went from 500+ms to 50ms) 🚀. Something along the lines of this is what I'm proposing:

# frozen_string_literal: true
module ActiveStorage

  class BlobsController < BaseController
    include ActiveStorage::SetBlob

    def show
      if ActiveStorage.redirect_to_service_url
        expires_in ActiveStorage.service_urls_expire_in
        redirect_to @blob.service_url(disposition: params[:disposition])
      else
        stream_blob
      end
    end

    def stream_blob
      expires_in ActiveStorage.service_urls_expire_in, public: true
      response.headers['Content-Type'] = params[:content_type]
      response.headers['Content-Disposition'] = params[:disposition]

      @blob.download do |chunk|
        response.stream.write(chunk)
      end
    ensure
      response.stream.close
    end
  end
end


Users looking for image loading with a CDN could set this up by setting

app.config.active_storage.redirect_to_service_url = false
app.config.active_storage.service_urls_expire_in = 1.year

I just threw this together for a project I'm working on, but I would be happy to clean this up and make a proper pull request if the Rails team is open to the idea. There's a fair amount of people requesting this feature https://github.com/rails/rails/issues/31419 some of the solutions in this issue are directly linking to the service files, but streaming would keep the service provider abstracted away while also allowing for proper CDN caching.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
--
Jonathan Fleckenstein

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.