disallow devise paths in robots.txt ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

disallow devise paths in robots.txt ?

fugee ohu
I see bots crawling devise pages in my log:
141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] "GET /users/sign_in HTTP/1.1" 200 4022 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
Should I disallow those paths in robots.txt ?

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/5091e20f-ec00-4764-9bb3-9a29a9bcbbf9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: disallow devise paths in robots.txt ?

Hassan Schroeder-2
On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu <[hidden email]> wrote:
> I see bots crawling devise pages in my log:

> Should I disallow those paths in robots.txt ?

1) why?

2) do your logs show these bots *reading* robots.txt?

--
Hassan Schroeder ------------------------ [hidden email]
twitter: @hassan
Consulting Availability : Silicon Valley or remote

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CACmC4yBoWsK3rHNEHckW5vTenKG4HWh-xUTKrtgepK274VBo3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: disallow devise paths in robots.txt ?

fugee ohu


On Monday, October 8, 2018 at 12:18:27 AM UTC-4, Hassan Schroeder wrote:
On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="m5aaGoZvAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:
> I see bots crawling devise pages in my log:

> Should I disallow those paths in robots.txt ?

1) why?

2) do your logs show these bots *reading* robots.txt?

--
Hassan Schroeder ------------------------ <a href="javascript:" target="_blank" gdf-obfuscated-mailto="m5aaGoZvAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">hassan.s...@...
twitter: @hassan
Consulting Availability : Silicon Valley or remote

Yes 

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/1fe76e02-c222-450b-8eca-16e4df491f18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.