{"id":28738,"date":"2022-03-31T06:00:00","date_gmt":"2022-03-31T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=28738"},"modified":"2022-04-01T09:42:34","modified_gmt":"2022-04-01T16:42:34","slug":"building-an-e-commerce-scraper","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/03\/31\/building-an-e-commerce-scraper\/","title":{"rendered":"Building an E-commerce Scraper"},"content":{"rendered":"\n

Sponsored Post<\/em><\/p>\n\n\n\n

Web scraping has undergone various changes and transformations ever since it rose into public consciousness. As automated data collection becomes more accessible over time, the customer base has increased, necessitating changes to accommodate the influx of people.<\/p>\n\n\n\n

We sit down with Aleksandras \u0160ul\u017eenko, Product Owner at Oxylabs.io<\/a>, to discuss the past, present, and future of web scraping. He has been involved with both the technical and business side of automated collection for several years and has helped create some of the leading web scraping solutions on the market such as the E-commerce Scraper API, dedicated to collecting external publicly available data from e-commerce websites.<\/p>\n\n\n\n

Over the years, how has your understanding and perception of the process changed? How has the industry evolved since you began working with web scraping?<\/strong><\/p>\n\n\n\n

I began my work when scraping was still niche. Today, if you have an online presence, you\u2019ll be somehow involved with scraping. You might be indirectly involved by using databases that have been compiled with web scraping, but you\u2019re still involved.<\/p>\n\n\n\n

Another important change, that is somewhat related, is the proliferation of online businesses. Back in the day, monitoring 50 pages was enough to cover the necessary grounds. Now you\u2019d need hundreds if not thousands of domains to get started.<\/p>\n\n\n\n

What also comes to mind is the technical side of scraping – it has become significantly harder. Anti-bot solutions, especially general ones, have risen in popularity. Proprietary solutions haven\u2019t been that much of an issue, though.<\/p>\n\n\n\n

Some of the solutions are fairly rudimentary. It\u2019s popular, for example, in Brazil to block any IP addresses coming from Europe. Systems such as these are incredibly easy to implement. But if you want to track bots efficiently, things get complicated quickly.<\/p>\n\n\n\n

All you get, server-side, is traffic. How do you differentiate between bots and humans? Almost all methods used to track the former lead to probabilistic evaluations. No matter how good the system is. Any bot-activity threshold set will eventually target a legitimate user.<\/p>\n\n\n\n

Usually, where the threshold is set depends on the non-technical decision makers of the business. Bot protection is always a trade-off between anti-scraping and user experience, which is, in the end, a trade-off for revenue. And you know what happens when there\u2019s danger to the bottom line.<\/p>\n\n\n\n

So, many businesses that have the technical capabilities of creating severe anti-bot protections don\u2019t do it, because even a minor impact on UX would have a sizable impact on user experience. They\u2019d rather only ban the really egregious examples of bots.<\/p>\n\n\n\n

In the end, it\u2019s a cat-and-mouse game with the deck stacked slightly in the favor of the mouse. Scraping is, I\u2019d rate, easy-to-medium while bot protection is hard.<\/p>\n\n\n\n

When the Scraper APIs were still in their early stages, what were the first or most important upgrades you implemented? <\/strong><\/p>\n\n\n\n

One of the greatest and most important updates has been the headless browser mode. It does cost more computing power overall than simple requests, but it lets you load JavaScript elements. Without them, lots of data wouldn\u2019t even be displayed and be otherwise impossible to reach.<\/p>\n\n\n\n

There are other ways to do the same process, such as through APIs, but they\u2019re often less scalable. Additionally, page URLs are much more static representations of specific places within a domain. It\u2019s unlikely that the same URL would start pointing to a completely different page after some time.<\/p>\n\n\n\n

In summary, a headless browser is more costly to run (when compared to sending requests), but it\u2019s more scalable than the other options and provides access to data that otherwise wouldn\u2019t be available.<\/p>\n\n\n\n

Our development of the Universal scraper has been a turning point as well. Previously, and I think like most starting out, we only created dedicated \u201cone-trick-ponies\u201d that worked on a single domain or, possibly, a set of pages.<\/p>\n\n\n\n

Universal scraping is a big step mainly because you can always turn it into a dedicated one with some settings. In other words, if a universal scraper can acquire data from 90% of existing web pages, you can always tweak it to fit some specific use case. <\/p>\n\n\n\n

You can\u2019t do it as easily the other way around – turn a dedicated scraper into a universal one. It\u2019s not quantitatively but qualitatively different.<\/p>\n\n\n\n

Would you start with dedicated Scraper APIs now over a more generic solution?<\/strong><\/p>\n\n\n\n

The decision on whether to start with a more generic or dedicated solution isn\u2019t the call of the tech people. Business needs take priority – if you have a deal for a single website, there\u2019s no reason to build a universal solution. But if you get a deal for hundreds, such a scraper might come in handy. It becomes extra valuable once you start thinking in the long-term.<\/p>\n\n\n\n

We should also take into account that data isn\u2019t distributed evenly across websites. There are certain center points where valuable information for some industries congregates. Scraping these tech giants might be more valuable than some accidental website of one small company.<\/p>\n\n\n\n

If you had to completely start over from a basic e-commerce scraper prototype, what would you develop and upgrade first?<\/strong><\/p>\n\n\n\n

One thing I should note is that regardless of what approach you take, what your business needs are, what are the resources available – every solution has to be scalable. I don\u2019t even recommend starting with anything if the solution isn\u2019t, at some point, going to become scalable.<\/p>\n\n\n\n

Web scraping is based on a complicated microservice architecture. You can\u2019t predict what exactly you\u2019ll need to scale ahead of time, but you\u2019ll definitely need to do it. Needs, desires, and capabilities will change, necessitating horizontal scaling at the very least<\/em>.<\/p>\n\n\n\n

There are two ways to talk about Scraper APIs if we\u2019re thinking of them at a higher level. One is the technical side of things – the scrapers, the parsers, the domains, etc. The other is the interaction (or the API) itself – how it takes, processes, and handles requests by users. These can be thought of as two distinct but closely related domains.<\/p>\n\n\n\n

The API is a little bit easier. We\u2019ve been developing application programming interfaces for ages. There\u2019s plenty of best practice manuals, development guides, and other sensible writings out there. I don\u2019t think we should be reinventing the wheel here, outside of the service side of the API.<\/p>\n\n\n\n

On the technical side, I\u2019d like to make things less complicated. As web scraping is going from niche to something everyone does, we\u2019d need fully visual tools with a variety of output formats. We had previously focused on JSON and HTML, however, many businesses have needs for other, differently structured output formats.<\/p>\n\n\n\n

Finally, we should be moving towards no-code formats to be more in tandem with the growing market. Researchers, entrepreneurs, and many other non-tech-savvy people are banging at the gates. We should be letting them in, instead of restricting access to web scraping through complicated technical layers.<\/p>\n\n\n\n

Oxylabs is carrying some good news as <\/em>Scraper APIs<\/em><\/a> are now available for a free trial! Now you can test them out and see them in action. Fetch public data from any site you like during a <\/em>one-week trial period<\/em><\/strong>.<\/em><\/p>\n\n\n\n