E-commerce

How to audit your e‑commerce store with Python and Scrapy?

How to audit your e‑commerce store with Python and Scrapy?

October 29, 2026

You are wondering how to audit your e‑commerce store with Python and Scrapy?

In short : in just a few lines of code you get the complete list of your pages, detect errors, and identify optimization levers.

This approach avoids expensive tools while giving you total control over the collected data, which is essential for improving SEO, speed, or accessibility.

So, how to audit your e‑commerce store with Python and Scrapy? On the agenda:

  • Why create your own crawler instead of buying a service?

  • What are the technical prerequisites to launch the project?

  • How to install Python and Scrapy on your server?

  • How to define and structure Items to extract URLs?

  • How does Qstomy complement this approach and simplify tracking?

Let's go.

In addition to inventory generation, the crawler allows you to extract metrics such as average load time or the depth of each page, giving you a global view of your site's technical health. Thanks to native integration with Qstomy APIs, you can automate the upload of these metrics into your periodic reports, thus facilitating data‑driven decision‑making.

This method adapts equally well to small catalogs and large multi‑vendor platforms, offering maximum flexibility for any type of online store.

Summary

Why create your own crawler instead of buying a service?

A home‑grown crawler gives you total freedom: you choose which pages to scan, the execution frequency, and the report format. Turnkey services often charge monthly licenses and do not allow access to raw data. By developing your own Python script, you can integrate it into your Qstomy pipeline, trigger automatic alerts, and enrich your dashboard with the indicators that truly matter for your store.

Moreover, the cost of a basic server and a few hours of development remains significantly lower than an annual subscription. This solution is therefore ideal for merchants who want to optimize their SEO, as explained in How SEO works for e‑commerce sites? .

By creating your own crawler, you can also customize filters to exclude admin pages, user accounts, or any other sensitive segments, ensuring the crawl does not disrupt your infrastructure. You also have the option to enrich each URL with metadata specific to your business, such as product type or category, which greatly simplifies post‑crawl analysis.

Convert over 2,000 customers on average per month with Qstomy.

The world’s 1st Shopify AI dedicated to customer conversion

Empowering 200+ e-commerce merchants

What are the technical prerequisites to launch the project?

Before coding, ensure you have a server (or VPS) with SSH access. You must be able to install Python packages and execute command‑line instructions from a terminal. An account with administrative rights is recommended to install system dependencies. Finally, prepare a dedicated directory where you will store your Scrapy project and output files.

If you are a beginner, start by installing an e‑commerce platform suited to your needs like Shopify, which integrates naturally with Python scripts via its API.

Also check for a valid HTTPS connection on your domain, as Scrapy follows secure redirects by default. Once the server is ready, it is wise to allocate sufficient disk space for logs and exports, especially if you plan daily crawls that can generate several megabytes of data.

Do not forget to configure the firewall to allow outgoing requests on port 443, which is essential for Scrapy to navigate HTTPS without blocking.

How to install Python and Scrapy on your server?

On most Linux distributions, Python 3 is already present. Check with python3 --version. Otherwise, install it via the package manager: sudo apt‑get install python3. Then, create a virtual environment to isolate dependencies: python3 -m venv venv && source venv/bin/activate. With the environment activated, install Scrapy: pip install scrapy. On macOS, use brew install python then follow the same steps. The official Scrapy documentation provides detailed examples.

Once installation is complete, quickly test the configuration with scrapy version to ensure Scrapy responds correctly. You can also install useful extensions like scrapy-splash for JavaScript rendering or scrapy-redis for distributing crawls across multiple workers, depending on the complexity of your store.

How to initiate a new Scrapy project?

Once Scrapy is installed, create a working directory and run the following command: scrapy startproject boutique. This instruction generates the standard folder structure (folders spiders, items.py, settings.py, etc.). You can immediately see the file scrapy.cfg which links your project to the interpreter. Rename the folder boutique as you prefer, for example audit, to reflect the script's mission.

After creating the project, remember to adjust the file settings.py : set ROBOTSTXT_OBEY = True to respect the robots.txt rules, configure DOWNLOAD_DELAY to avoid overloading the server, and enable the export pipeline if you wish to directly generate JSON or CSV files. These settings ensure a polite and reliable crawl.

To handle potential blocks, activate the proxy rotation middleware in settings.py ; this allows distributing requests across multiple IP addresses and avoiding server-side limitations.

How to automatically generate a CrawlSpider?

Scrapy offers a spider generator. In the same directory, execute: scrapy genspider -t crawl monspider exemple.com. The flag -t crawl creates a CrawlSpideralready equipped with a LinkExtractor that follows links. The file monspider.py appears under spiders/. You simply need to specify the target domain (example.com) and the entry point (the homepage).

To refine the behavior, you can edit the generated class and add additional rules, such as URL restrictions with allow ou deny. You can also inject custom headers (User-Agent, Accept-Language) to simulate different browsers and reduce the risk of server blocking.

How to define Items to retrieve URLs?

In the file items.py, create a class that inherits from scrapy.ItemAdd a field url = scrapy.Field()It looks like:
class PageItem(scrapy.Item): url = scrapy.Field()

This structure allows storing each discovered URL in a structured object, facilitating later export (JSON, CSV, etc.). You can add other fields (title, HTTP status) when the need to analyze more finely arises.

For example, to retrieve the HTTP status code, simply add status = scrapy.Field() and fill it in your parsing method. This will give you an overview of pages returning 404 or 500 errors, crucial for monitoring site health.

How to customize link extraction rules?

In the spider, the variable rules determines which links will be followed. To retrieve *all* links, simplify the rule: Rule(LinkExtractor(), callback=’parse_page’, follow=True). This instruction tells Scrapy to extract each href present on the current page, call the function parse_page for each response, and then continue recursive navigation.

If you need to exclude URLs (e.g., admin pages), use the argument deny of the LinkExtractor with a suitable regular expression.

It is also possible to combine multiple rules to apply different treatments depending on the page type (product, category, blog). For example, one rule can call parse_product for product pages, while another uses parse_category for category lists.

How to extract the URL of each visited page?

In the method parse_pagecreate an instance of your Item puis affectez response.url in the field url. Exemple :
def parse_page(self, response): item = PageItem() item[’url’] = response.url return item

No other processing is required; the spider will simply return each URL encountered. You can enrich the object with other information, such as the status code (response.status) or the page title (response.xpath(’//title/text()’).get()).

To capture the server response time, add item[’load_time’] = response.meta[’download_latency’]. This data becomes very useful for identifying slow pages and guiding performance optimizations.

How to launch the crawl and save results?

Navigate to the project directory and execute: scrapy crawl monspider -o pages.json. L’option -o tells Scrapy to write items to the file pages.json in JSON format. You can also choose -o pages.csv for an Excel-compatible array.

To automate the process, create a cron task that launches the crawl every night. Thus, you will have a daily updated inventory of your store's URLs, usable in your Qstomy reports.

If you wish to run the crawl in verbose mode to diagnose potential issues, add the flag -s LOG_LEVEL=DEBUG. You can also redirect the output to a log file to maintain a historical record of executions and any errors encountered.

The log file generated by Scrapy can be sent to solutions like Datadog or Grafana, offering a real-time visual dashboard to quickly detect errors or crawl slowdowns.

How to analyze data to identify optimization points?

Import the file pages.json into an analysis tool (pandas, Excel, or Google Sheets). Look for duplicate URLs, 4xx/5xx responses, and orphaned pages (without inbound links). These signals indicate SEO or user experience issues.

By cross-referencing results with notre guide d’optimisation Google, you can prioritize fixes to apply. Then export the lists of pages to correct and import them into Qstomy to plan development actions.

Once anomalies are identified, you can automate the generation of JIRA or Asana tickets directly from Qstomy using integrated webhooks, ensuring that every detected issue is tracked until resolution.

How does Qstomy facilitate your store audit using Python and Scrapy?

Qstomy relies on your crawls to feed a centralized dashboard. After each spider execution, you can upload the JSON file via the Qstomy interface; the system transforms it into key metrics: number of pages, error rate, and average load time. You then receive automatic recommendations, such as updating meta tags alt or reducing image file sizes.

Additionally, Qstomy triggers email alerts or in-app notifications as soon as an anomaly exceeds the defined threshold. This gives you continuous monitoring without needing to develop additional scripts.

The dashboard also offers interactive visualizations (response time heatmaps, HTTP code distribution charts) that help product and technical teams quickly identify critical areas for optimization.

What checklist before deploying your crawler in production?

In short

  • Serveur stable : check disk space and CPU load.

  • Permissions SSH : ensure the user has the necessary permissions.

  • Version Python : use Python 3.8+ for Scrapy compatibility.

  • Tests locaux : run the spider in scrapy crawl --nolog mode to validate the logic.

  • Qstomy integration : schedule automated import of results into the dashboard.

FAQ

  • Q: Does the crawler impact site performance? A: The crawl is limited to a reasonable speed (default 8 requests/s) to avoid overloading the server.

  • Q: Can I filter sections of the site? A: Yes, use the parameters deny ou allow of the LinkExtractor.

To go further: How to create a payment gateway for an e-commerce site? - Qstomy, How to drive traffic to an online store (SEO, ads, social media) - Qstomy, Why use social media for e-commerce customer service? - Qstomy, How to use an AI chatbot to compare two products in your store? - Qstomy.

Finally, remember to add health monitoring for the crawler itself by recording the total crawl duration and the number of pages processed. These indicators will allow you to adjust execution frequency and detect potential slowdowns related to the growth of your catalog.

Enzo

October 29, 2026

Convert over 2,000 customers on average per month with Qstomy.

The world’s 1st Shopify AI dedicated to customer conversion

Empowering 200+ e-commerce merchants

Subscribe to the newsletter and get a personalized e-book!

No-code solution, no technical knowledge required. AI trained on your e-shop and non-intrusive.

*Unsubscribe at any time. We do not send spam.

Subscribe to the newsletter and get a personalized e-book!

No-code solution, no technical knowledge required. AI trained on your e-shop and non-intrusive.

*Unsubscribe at any time. We do not send spam.