Beezlabs Innovation Labs | Appointments Naviagtion Drawer

Web crawling for competitive pricing analysis

image background-image
1 Mn+
Monthly Records
1000 +
Efforts Savings (Hrs)
Websites Being Scraped

Client portfolio

Our client is one of the largest liquor manufacturing companies in the United States of America, with a perfect blend of rich ancestral heritage and modern innovative flavor, working strenuously at finding the right balance between staying ahead of customers’ expectations and delivering quality experiences.

Overview – Origins

A recent research showed that, about 80% of buyers of all products compare the offers of several stores in search of a better bargain. For a retailer or a manufacturer to make sure that his product is chosen by the buyers, he has to analyze the market, study the pricing of his competitors, map his real position in terms of pricing against his competitors and then reach a pricing strategy. The end goal is to reach an optimal pricing wherein the retailer’s products grab the attention of the buyers in terms of its competitive prices and at the same time ensure a decent profit for the retailer.

Businesses employ the strategy of Competitive pricing whose primary goal is to attract more customers by optimizing prices and thereby increasing sales and revenue. For this strategy, the legal extraction of pricing and sales data of the competitors from various websites is done.

One such site is Drizly, an online marketplace for alcohol is North America which partners with thousands of retailers and offers a rich e-commerce shopping experience.

Problem statement – The Past

As part of an efficient sales strategy for their products, our client performs Data Analysis on the competitors’ products on a regular basis. For this purpose a dedicated individual extracts data manually from various e-commerce portals, one of which is Drizly. For such extraction he has to login to the Drizly site and provide a Store address. Drizly displays all products with the rating, quantity and price. The user now has to download all this data on a record by record basis. Once the data is completely downloaded, he pushes the same to a centralized location for further analytics.

Drizly deals with huge volumes of data, in terms of multiple products in multiple measures across multiple stores. Therefore a manual extraction is time-consuming, not to mention monotonous and also challenging. There is always the margin of error as in with any manual process.

Our Solution – The Present

The BeezLabs-developed service SmartScraper has the capability to comb through websites and fetch desired data. Therefore SmartScraper was user to crawl through the Drizly website, with the Store address as an input. Another input would be the location where the extracted data has to be stored. SmartScraper working in close integration with Tulip enables the necessary data files to be downloaded from the Drizly site in a totally touchless manner. The extracted files are now uploaded to the centralized location as specified in the input. Further metrics on the number of records fetched and the average bot run time are updated for the clients’ perusal in PowerBi.

This crawling can be configured to be run on a regular basis, at a specified frequency. So, there you have it! SmartScraper has minimized the overhead to manually having to open sites and look for products data record by record.

Value delivered – The Outcome

Faster turn-around – From days and hours to minutes and seconds
On demand anytime real time data
Broadens the horizon and efficacy of Analytics by fetching any kind of data from any number of sources, hence better sales plan and increased revenue
No more manual errors or predictions on ambiguity required

Potential extended scope – The Future

In today’s fast moving competitive world, it is not just essential to have analyzed the market status of today but also to be able to predict the status of the future. With SmartScraper, by customizing the sites to be crawled and the data to be fetched, a whole new take on competitive analysis could be achieved. The extension can be done from pricing to demand analysis to customer expectations analysis and so on. The applications of SmartScraper are limitless.