Skip to main content

Agentic SEO

Technical SEO Audit with AI Agents

Visit Websitetonif95/TechnicalSEOAgent
Agentic SEO

This advanced system has been designed to automate and simplify the technical SEO audit process for websites, providing users with detailed reports on issues affecting their visibility in search engines. Leveraging a modern architecture and the power of artificial intelligence, the system offers a comprehensive solution from initial crawling to generating actionable reports.

System Architecture

The system is based on a well-defined three-layer architecture, ensuring scalability, modularity, and a clear separation of responsibilities:

  • Frontend: User Interface (UI) built with React.

  • Backend: RESTful API developed with FastAPI.

  • AI Agents: Intelligent modules powered by the OpenAI Agents SDK, responsible for crawling, analysis, and report generation.

  • Database: SQLite for persistent storage of crawled and analyzed data.

Key Components

Frontend (React)

The frontend is the visible face of the system and the main interaction point for the user.

Technology

Developed with React, enabling a dynamic, responsive, and efficient user interface.

Functionality

  • Allows the user to input the website URL to audit and the maximum number of pages to crawl.

  • Manages the user’s OpenAI API key input securely (stored locally).

  • Displays the status of the process (crawling, generating report).

  • Presents the generated technical SEO report in a readable and navigable format.

  • Offers the option to clear previous crawl data.

  • Interaction with Backend: Communicates with the FastAPI backend via HTTP requests (POST to start crawling and generate report, DELETE to clear the database).

Backend (FastAPI)

The backend acts as the brain of the system, orchestrating operations and serving as an intermediary between the frontend and the AI agents.

Technology

Built with FastAPI, a modern, high-performance web framework for building APIs in Python.

Functionality

  • API Endpoints: Exposes entry points for system operations (/crawl/, /generate-report/, /clear-database/).

  • Crawl Orchestration: The /crawl/ endpoint receives the URL and page limit from the frontend. It is crucial that this endpoint blocks its response until the crawling process has fully completed. This is achieved by running the crawl logic (run_crawl_process_sync) in a separate thread using asyncio.to_thread(), ensuring FastAPI waits for its completion before sending confirmation to the frontend.

  • API Key Management: Receives the OpenAI API key from the frontend and sets it as an environment variable (OPENAI_API_KEY) so the AI agents can authenticate.

  • Report Orchestration: The /generate-report/ endpoint starts a separate process to run the analyzer agent, waiting for its result before returning the report to the frontend.

  • Database Management: Directly interacts with the SQLite database to store and retrieve crawled data. The /clear-database/ endpoint deletes all crawl records.

  • Concurrency: Uses asyncio.to_thread to handle blocking operations (such as HTTP requests from the crawler or AI agent execution) without blocking FastAPI’s main event loop, keeping the API responsive.

Agent System (OpenAI Agents SDK)

The intelligent core of the system lies in the AI agents, automating complex SEO tasks.

Technology

Implemented using the OpenAI Agents SDK, which facilitates creating and managing conversational and autonomous agents.

Main Agents

Crawler Agent (implicit in crawler.py):

  • Role: Responsible for browsing the website, fetching the HTML content of pages, and extracting internal links.

  • Process: Uses requests for HTTP requests and BeautifulSoup for HTML parsing.

  • Output: Provides the original HTML, prettified HTML, and links found for each URL.

  • Analyzer Agent (in analyzer.py):

  • Role: Receives crawled HTML data and performs a detailed technical SEO analysis. Identifies issues such as missing titles, meta descriptions, robots tags, canonicalization problems, header structure (H1, H2), internal/external links, image usage (alt text, lazy loading), and JavaScript and structured data aspects.

  • Process: Uses AI capabilities of the OpenAI Agents SDK to interpret data and generate findings and recommendations.

  • Output: Produces a structured report with key findings, potential causes, SEO impact, and practical recommendations.

Strategist Agent (in strategist.py):

  • Role: Takes the technical report generated by the Analyzer Agent and transforms it into a prioritized strategic action plan.

  • Process: Uses the OpenAI Agents SDK to translate technical findings into clear, actionable steps for a non-technical user, estimating impact and implementation effort.

  • Output: Generates a detailed, easy-to-follow SEO action plan.

Database (SQLite)

Technology

SQLite is used, a lightweight, file-based database ideal for this type of application.

Functionality

Persistently stores the original HTML, prettified HTML, and SEO analysis results for each crawled page. This allows the agent system to access data for report generation and preserves data between runs if needed (though data is cleared after the report).

System Workflow

  • Crawl Start: The user inputs a URL and number of pages in the frontend (React) and clicks "Start Crawl."

  • Backend Request: The frontend sends a POST request to /crawl/ in the backend (FastAPI), including the URL, page limit, and OpenAI API key.

  • Blocking Crawl: The /crawl/ endpoint in FastAPI starts the run_crawl_process_sync function in a separate thread (asyncio.to_thread). This thread is responsible for:

  • Crawling the website pages, respecting the specified limit.

  • For each page, fetching the HTML and passing it to analyze_html_content (which may internally use an AI agent for analysis).

  • Saving the analysis results in the SQLite database.

  • Continuing the crawl loop until the max_pages number of pages have been successfully analyzed and stored.

  • Crawl Confirmation: Once run_crawl_process_sync completes (i.e., crawling and data saving have finished), the /crawl/ endpoint sends a 200 OK response to the frontend.

  • Report Generation: Upon receiving crawl confirmation, the frontend calls the /generate-report/ endpoint on FastAPI.

Analyzer Agent Execution: The backend starts a separate process to run _generate_report_in_process (which in turn calls generate_technical_seo_report and uses the Analyzer Agent). This agent reads the analysis data from the database, processes it, and generates the technical SEO report.

  • Report Display: The generated report is returned to the frontend and displayed to the user in a modal.

  • Database Cleanup: After generating and showing the report, the frontend sends a DELETE request to /clear-database/ to clear the database, ensuring current audit data does not interfere with future audits.

System Benefits

  • Full Automation: Automates the technical SEO audit process, reducing manual effort.

  • Detailed Reports: Generates comprehensive reports with clear findings and recommendations.

  • Actionability: Transforms complex technical data into easy-to-follow action plans.

  • Cutting-Edge Technology: Uses FastAPI for high-performance backend and OpenAI Agents SDK for advanced AI capabilities.

  • Intuitive User Experience: The React frontend provides a clean and user-friendly interface.

  • This system represents a powerful tool for any professional or website owner looking to efficiently identify and solve technical SEO issues.