How do i web scrape aliexpress from rails (console) or app

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How do i web scrape aliexpress from rails (console) or app

fugee ohu
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/c02eace0-e2f0-4c44-b00d-791239230388%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

Colin Law
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CAL%3D0gLtqZHcSz%3DtfnzwrJrwUGbj%2Bed28262nSkL%3DhJ8yC1h%3D5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

fugee ohu


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="6aPbaeHxBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you? I can give you specific selectors, counts, text, etc

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/703241e0-3d4d-4502-bffd-8e87b207b84a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

fugee ohu
In reply to this post by Colin Law


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="6aPbaeHxBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


 https://www.aliexpress.com/wholesale?catId=0&initiative_id=SB_20180728063558&SearchText=home+robot

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/5610cee7-fe78-4d08-a100-11347f34b911%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

Colin Law
In reply to this post by fugee ohu
On 28 July 2018 at 15:32, fugee ohu <[hidden email]> wrote:


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you?

If you are now seeing the data then no I don't want to see it.  Your original problem is fixed.

Colin

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CAL%3D0gLuLHdryZk_hsK3fh7H%3DaG_KEjDHcSeaPav%2BAdAmvcheiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

fugee ohu


On Saturday, July 28, 2018 at 10:43:37 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 15:32, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="NXXF8IELBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you?

If you are now seeing the data then no I don't want to see it.  Your original problem is fixed.

Colin


I could read the item description on the page in the browser but couldn't find it searching the source I'm new to using the browser debugging console I never had any reason to use it before So I couldn't find the description I saw on the page in the source so I right clicked on the description and selected inspect This is what I got: 
<li class="property-item" id="product-prop-19204" data-attr="272" data-title="Wireless" data-spm-anchor-id="2114.12010108.0.i4.7c48412aEPVyt2">
                        <span class="propery-title">Cord Length (m):</span>
                        <span class="propery-des" title="Wireless">Wireless</span>
                    </li>
What can I do with this?

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/a73bb316-d26e-42c1-8abe-3dc1585f2fc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

Colin Law
On 29 July 2018 at 03:43, fugee ohu <[hidden email]> wrote:


On Saturday, July 28, 2018 at 10:43:37 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 15:32, fugee ohu <[hidden email]> wrote:


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you?

If you are now seeing the data then no I don't want to see it.  Your original problem is fixed.

Colin


I could read the item description on the page in the browser but couldn't find it searching the source I'm new to using the browser debugging console I never had any reason to use it before So I couldn't find the description I saw on the page in the source so I right clicked on the description and selected inspect This is what I got: 
<li class="property-item" id="product-prop-19204" data-attr="272" data-title="Wireless" data-spm-anchor-id="2114.12010108.0.i4.7c48412aEPVyt2">
                        <span class="propery-title">Cord Length (m):</span>
                        <span class="propery-des" title="Wireless">Wireless</span>
                    </li>
What can I do with this?

If you can see the description in the browser but not in the source then as a web developer I am sure you can think of at least a couple of ways that might happen.

Colin

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CAL%3D0gLtFmOHPoPUh%2Bk8sGrmO19cUxAOxHuUixmfsRHG708bs9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

fugee ohu


On Sunday, July 29, 2018 at 4:14:03 AM UTC-4, Colin Law wrote:
On 29 July 2018 at 03:43, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="OcHjXdREBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:


On Saturday, July 28, 2018 at 10:43:37 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 15:32, fugee ohu <[hidden email]> wrote:


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you?

If you are now seeing the data then no I don't want to see it.  Your original problem is fixed.

Colin


I could read the item description on the page in the browser but couldn't find it searching the source I'm new to using the browser debugging console I never had any reason to use it before So I couldn't find the description I saw on the page in the source so I right clicked on the description and selected inspect This is what I got: 
<li class="property-item" id="product-prop-19204" data-attr="272" data-title="Wireless" data-spm-anchor-id="2114. 12010108.0.i4.7c48412aEPVyt2">
                        <span class="propery-title">Cord Length (m):</span>
                        <span class="propery-des" title="Wireless">Wireless</span>
                    </li>
What can I do with this?

If you can see the description in the browser but not in the source then as a web developer I am sure you can think of at least a couple of ways that might happen.

Colin


It's a new subject for me I found a lot of window.runParams statements within script containers so I guess I'm gonna make those ajax requests myself from my script

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/ea205a41-465c-42c6-bf0a-7222fa85d2c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do i web scrape aliexpress from rails (console) or app

fugee ohu
In reply to this post by Colin Law


On Sunday, July 29, 2018 at 4:14:03 AM UTC-4, Colin Law wrote:
On 29 July 2018 at 03:43, fugee ohu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="OcHjXdREBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fuge...@...> wrote:


On Saturday, July 28, 2018 at 10:43:37 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 15:32, fugee ohu <[hidden email]> wrote:


On Saturday, July 28, 2018 at 2:53:59 AM UTC-4, Colin Law wrote:
On 28 July 2018 at 00:15, fugee ohu <[hidden email]> wrote:
how do i webscrape aliexpress? i learned how to get a document object class Nokogiri::HTML containing the full html but there's no tables only ul and li selectors and i'm having trouble with that

Give us an example of a page you are scraping and an element you can't find.

Colin


Thanks Colin Made progress so now I have the <ul> list i want in an object I named ul Not sure how I'm gonna proceed My goal is to import to database tables  It's a few related tables instead of just one products table I don't think you still wanna see all that html do you?

If you are now seeing the data then no I don't want to see it.  Your original problem is fixed.

Colin


I could read the item description on the page in the browser but couldn't find it searching the source I'm new to using the browser debugging console I never had any reason to use it before So I couldn't find the description I saw on the page in the source so I right clicked on the description and selected inspect This is what I got: 
<li class="property-item" id="product-prop-19204" data-attr="272" data-title="Wireless" data-spm-anchor-id="2114.12010108.0.i4.7c48412aEPVyt2">
                        <span class="propery-title">Cord Length (m):</span>
                        <span class="propery-des" title="Wireless">Wireless</span>
                    </li>
What can I do with this?

If you can see the description in the browser but not in the source then as a web developer I am sure you can think of at least a couple of ways that might happen.

Colin


Even if so, wouldn't the javascript leave the html visible in the source? 

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/becd3904-e3cd-46e2-9161-4d2f57766829%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.