E-commerce

How to track competitor prices with Python and Scrapy?

How to track competitor prices with Python and Scrapy?

October 31, 2026

Are you wondering how to monitor competitor prices without spending hours every day?

The solution involves a small Python script using the Scrapy framework: it explores target sites, extracts prices, and delivers ready-to-use data.

This approach requires some knowledge of HTML and CSS, but remains accessible to any motivated merchant looking to automate price tracking.

In addition to reducing time spent on competitive monitoring, automation provides continuous monitoring, allowing you to anticipate price drops or increases before your customers even notice.

So, how to track competitor prices with Python and Scrapy? Here's the plan:

  • Why is price tracking crucial for your Shopify store?

  • What are Scrapy's strengths compared to paid services?

  • How to prepare your environment: Python, pip, and dependencies?

  • How to create a Scrapy project and define the elements to extract?

  • How does Qstomy connect to your robot to transform data into concrete actions?

Let's go.

Before you begin, ensure your store complies with privacy policies and that you have clear permission to collect this public data. This will avoid any legal issues and ensure ethical monitoring.

Summary

Why is price tracking crucial for your Shopify store?

In a market where prices fluctuate daily, knowing your competitors' prices allows you to adjust your rates in real-time, avoid margin losses, and attract price-sensitive buyers. By quickly detecting price increases or decreases, you can schedule targeted promotions or take advantage of additional margins. Beyond competitiveness, price tracking feeds your trend analysis, helps you identify best-sellers, and anticipate stock-out seasons. In short, it is an essential profit lever for any Shopify merchant who wants to stay agile.

Furthermore, having a price history allows you to perform advanced economic modeling, such as forecasting price elasticity or the impact of seasonal variations on revenue.

Moreover, price tracking enables you to calculate key indicators such as the average segment price, the daily change rate, and relative market share. These KPIs feed your strategic dashboard and facilitate quick adjustment decisions.

Convert over 2,000 customers on average per month with Qstomy.

The world’s 1st Shopify AI dedicated to customer conversion

Empowering 200+ e-commerce merchants

What are Scrapy's strengths compared to paid services?

Solutions like Wiser or Price Trakker offer turnkey platforms, but their costs can quickly exceed the budgets of small shops. Scrapy, on the other hand, is a free, open-source, and highly customizable framework. It works on all systems (macOS, Linux, Windows) and requires no dedicated server: your computer is enough. Additionally, you have full control over crawling frequency, retrieved fields, and output format (CSV, JSON, database). This freedom allows you to directly integrate data into Qstomy flows, automate price adjustments, or feed your analysis dashboard.

Furthermore, the active community around Scrapy offers numerous plugins (middlewares, extensions) that facilitate proxy management, compliance with robots.txt rules, and throttling to avoid banning target sites.

Among the most useful extensions are scrapy‑proxy‑pool to automatically manage IP addresses, scrapy‑autoextract which simplifies the extraction of structured data, and scrapy‑crawlee which adds support for JavaScript sites via Playwright.

How to prepare your environment: Python, pip, and dependencies?

Before coding, ensure you have:

  • Python 2.7 or higher installed.

  • The package manager pip.

  • The libraries setuptools, lxml, and OpenSSL.

On macOS or Linux, Python is often present by default; you just need to add pip with the command sudo python get-pip.py, then update the package six (required for Scrapy) via sudo pip install --upgrade six. Finally, install Scrapy with sudo pip install Scrapy. On Windows, follow the detailed procedure on the official Scrapy website.

Also consider creating a virtual environment (via python -m venv env) to isolate your project dependencies and easily reproduce the same setup on other machines.

How to create a Scrapy project and define the elements to extract?

Open your terminal, navigate to the working directory, and run:

scrapy startproject prix_concurrents

This command generates the following folder structure:

prix_concurrents/
├─ scrapy.cfg
├─ prix_concurrents/
│  ├─ __init__.py
│  ├─ items.py
│  ├─ pipelines.py
│  ├─ settings.py
│  └─ spiders/

The items.py is used to declare the fields you want to retrieve, for example nom_produit, prix and url. You add them in the form of scrapy.Field().

You can also create a file middlewares.py to manage HTTP headers, request delays, or proxy rotation to remain invisible to anti-scraping systems.

How to write the spider that traverses the category and retrieves prices?

In spiders/, create a file tfaw_spider.py. Voici l’essentiel :

import scrapy
from prix_concurrents.items import PrixConcurrentsItem

class TFAWSpider(scrapy.Spider):
    name = ’tfaw’
    start_urls = [’https://www.tfaw.com/collections/mass-effect’]

    def parse(self, response):
        for link in response.css(’a.product-link::attr(href)’).getall():
            yield response.follow(link, self.parse_product)

    def parse_product(self, response):
        item = PrixConcurrentsItem()
        item[’nom_produit’] = response.css(’h1.title::text’).get().strip()
        item[’prix’] = response.css(’span.price::text’).re_first(r’\d+[.,]?\d*’)
        item[’url’] = response.url
        yield item

The spider first retrieves product page URLs from the category page, then, for each product, extracts the name, price, and address. You can adjust the CSS selectors according to the target site's structure.

Don't forget to add a method custom_settings to set the delay between requests (DOWNLOAD_DELAY) or enable respect for the robots.txt, making the crawler more respectful of the site's policies.

How to handle pagination to cover the entire catalog?

Many sites display their products across multiple pages. Simply add a link tracking for the 'next' button:

def parse(self, response):
    for link in response.css(’a.product-link::attr(href)’).getall():
        yield response.follow(link, self.parse_product)
    next_page = response.css(’a.next::attr(href)’).get()
    if next_page:
        yield response.follow(next_page, self.parse)

The spider continues as long as a 'next' button is present, ensuring exhaustive price collection.

For sites using infinite scroll, you can replace this mechanism with an internal API or simulate dynamic loading with Selenium, but this increases complexity and execution time.

How to export the data to a usable format?

Scrapy offers direct export pipelines: run the crawl with the option -o prix.csv ou -o prix.json. Exemple :

scrapy crawl tfaw -o prix.csv

The generated file contains one line per product with the fields defined in items.py. You can then import it into your Qstomy dashboard or connect it to a database via a custom pipeline.

If you need a richer format (e.g., MongoDB or PostgreSQL), create a pipeline in pipelines.py which opens the connection to the database and inserts each item on the fly, avoiding the storage of temporary files.

How to automate the script execution daily?

On macOS or Linux, create a cron job:

0 3 * * * cd /chemin/vers/prix_concurrents && scrapy crawl tfaw -o prix_$(date +\%Y\%m\%d).csv

This line runs the spider at 3 AM, generates a dated file, and retains history. On Windows, use Task Scheduler with the same command. Automation guarantees a continuous data stream, ready to be leveraged by Qstomy.

In addition to cron, you can leverage services like GitHub Actions or AWS Lambda to trigger the crawl from the cloud, freeing you from managing a dedicated server.

How to integrate the retrieved prices into Qstomy recommendations?

Qstomy can read CSV files via its import module: choose 'Import of competitive data' in the dashboard, upload the generated file, and then map the columns nom_produit, prix and url. Once the data is assimilated, Qstomy offers:

  • Automatic price adjustments based on the market average.

  • Alerts when a competitor lowers their price by more than 5%.

  • Recommendations for bundles or targeted promotions for underperforming products.

The flow remains fully configurable, giving you control while benefiting from Qstomy's intelligence.

You can also configure advanced mapping rules, for example, applying a multiplier coefficient to imported prices to maintain a target margin before Qstomy automatically adjusts them.

How to leverage price history to anticipate trends?

By retaining each daily export, you build a time series. Import these archives into your analysis tool (Google Sheets, Power BI, or the Qstomy analysis module). Track the average price evolution by category, identify seasonal peaks and recurring drops. These insights help you plan your sales campaigns, negotiate purchases in advance, and offer attractive prices at the right time.

Thanks to regression functions or time series models (ARIMA, Prophet), you can even forecast future variations and prepare dynamic pricing scenarios that automatically adapt to market conditions.

How does Qstomy help transform Scrapy data into concrete actions?

Qstomy acts as a bridge between your price data and your conversion goals. After import, it:

  • Analyzes price gaps and suggests dynamic adjustments.

  • Automatically updates your Shopify product page prices via its integrated API.

  • Enriches upsell/cross-sell recommendations based on top-performing products on the competitor side.

  • Generates weekly reports and alerts your team when a competitor launches a major promotion.

Thus, the scraping work becomes immediately actionable without manual manipulation.

Additionally, Qstomy allows creating A/B test scenarios where different discount levels are automatically applied based on competitor prices to optimize conversion rates in real time.

Quelle checklist avant de lancer son robot de suivi de prix ?

Pre-launch checklist

  • Verify the legality of scraping for each target site (terms of use).

  • Identify exact CSS/XPath selectors for name, price, and URL.

  • Test the spider on a single page before launching the full crawl.

  • Schedule execution frequency (cron or Windows Task Scheduler).

  • Configure automatic import in Qstomy and verify field mapping.

In short

Scrapy offers an economical, flexible, and powerful solution for tracking competitor prices. Coupled with Qstomy, it transforms every data point into a sales opportunity.

FAQ

  • Does Scrapy block secure sites? Yes, provided the SSL certificate is valid and you are using the latest version of the tool.

  • Can I scrape multiple sites with the same project? Simply create multiple spiders in the directory spiders/each with its start_urls dedicated.

  • What data format does Qstomy prefer? CSV and JSON are the simplest; Qstomy accepts both via its integrated importer.

To go further: SEO for e-commerce sites: strategies that really work - Qstomy, How does SEO work for e-commerce sites? - Qstomy, Feedback loop: follow feedback to optimize your products - Qstomy, How to optimize an e-commerce site for Google (step-by-step guide) - Qstomy, E-commerce SEO strategy for category pages - Qstomy, How to build an e-commerce Facebook Ads strategy? - Qstomy, How to optimize a product page for conversion? - Qstomy.

Enzo

October 31, 2026

Convert over 2,000 customers on average per month with Qstomy.

The world’s 1st Shopify AI dedicated to customer conversion

Empowering 200+ e-commerce merchants

Subscribe to the newsletter and get a personalized e-book!

No-code solution, no technical knowledge required. AI trained on your e-shop and non-intrusive.

*Unsubscribe at any time. We do not send spam.

Subscribe to the newsletter and get a personalized e-book!

No-code solution, no technical knowledge required. AI trained on your e-shop and non-intrusive.

*Unsubscribe at any time. We do not send spam.