{"id":738,"date":"2024-10-09T16:02:01","date_gmt":"2024-10-09T14:02:01","guid":{"rendered":"https:\/\/www.syntera.ch\/blog\/?p=738"},"modified":"2024-10-09T16:02:02","modified_gmt":"2024-10-09T14:02:02","slug":"scrape-dynamic-apis-like-a-pro-with-selenium","status":"publish","type":"post","link":"https:\/\/www.syntera.ch\/blog\/2024\/10\/09\/scrape-dynamic-apis-like-a-pro-with-selenium\/","title":{"rendered":"Scrape Dynamic APIs Like a Pro with Selenium"},"content":{"rendered":"
<\/div>

LEAD DATA ENGINEERING<\/p>

Kail Kuhn Schlicht<\/p><\/div><\/div>\n\n\n

At Syntera, we\u2019re passionate about blending creativity with technology, especially when it comes to automation and data extraction. In this blog post, I\u2019ll show you how to scrape data from dynamic APIs like a pro<\/strong>, using Selenium<\/strong> with Firefox<\/strong>, a simple Django Ninja<\/strong> API, and a basic web page<\/strong>. You\u2019ll learn how to automate interactions on a web page, capture data from an API, and save it in a structured format\u2014all without manually parsing messy HTML.<\/p>\n\n\n\n

Use Case: Fetching Mountain Passes Data from a Local API<\/strong><\/h3>\n\n\n\n

Imagine you have a web application that provides information about Swiss mountain passes. The app features a button labeled “Load Passes”<\/strong>, which triggers an AJAX request to a \/api\/mountain-passes<\/code> endpoint to fetch the data. Your task is to:<\/p>\n\n\n\n

    \n
  1. Automate interaction with the web page<\/strong> to click the “Load Passes” button.<\/li>\n\n\n\n
  2. Capture the returned data<\/strong> from the dynamic API request.<\/li>\n\n\n\n
  3. Save the data<\/strong> in a structured format (JSON).<\/li>\n<\/ol>\n\n\n\n

    This post will guide you through setting up Django Ninja<\/strong> as a lightweight API for serving the mountain pass data, creating a sample web page<\/strong>, and then automating the data extraction process with Selenium<\/strong>.<\/p>\n\n\n\n

    Step 1: Setting Up the Django Ninja API<\/strong><\/h4>\n\n\n\n

    First, let\u2019s create a simple API using Django Ninja<\/strong>. Django Ninja is a powerful, fast, and modern web framework that makes it easy to build APIs with minimal code. For this use case, we\u2019ll set up a simple endpoint to return mountain pass information.<\/p>\n\n\n\n

    You can refer to the GitHub repository<\/a> for the complete code setup for the Django Ninja<\/strong> API, including the models and endpoints. Here\u2019s a quick overview:<\/p>\n\n\n\n

      \n
    1. Create a Django project and app<\/strong> named mountains<\/code>.<\/li>\n\n\n\n
    2. Set up Django Ninja<\/strong> in the app and define an API endpoint \/api\/mountain-passes<\/code> that returns a list of mountain passes.<\/li>\n\n\n\n
    3. Run the Django server locally<\/strong> on http:\/\/127.0.0.1:8000<\/code>.<\/li>\n<\/ol>\n\n\n\n

      Now, the Django Ninja<\/strong> API will serve the mountain pass data in JSON format when the \/api\/mountain-passes<\/code> endpoint is accessed.<\/p>\n\n\n\n

      Step 2: Creating a Sample Web Page with a “Load Passes” Button<\/strong><\/h4>\n\n\n\n

      We need a simple web page that contains a button to trigger the AJAX request to our API. Here\u2019s the HTML code for the sample web page:<\/p>\n\n\n\n

      <!DOCTYPE html>\n<html lang=\"en\">\n<head>\n    <meta charset=\"UTF-8\">\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n    <title>Mountain Passes<\/title>\n    <script src=\"https:\/\/ajax.googleapis.com\/ajax\/libs\/jquery\/3.5.1\/jquery.min.js\"><\/script>\n<\/head>\n<body>\n    <h1>Swiss Mountain Passes<\/h1>\n    <button id=\"load-passes\">Load Mountain Passes<\/button>\n    <ul id=\"passes-list\"><\/ul>\n\n    <script>\n        $('#load-passes').on('click', function() {\n            $.ajax({\n                url: '\/api\/mountain-passes',\n                method: 'GET',\n                success: function(data) {\n                    $('#passes-list').empty();\n                    data.forEach(function(pass) {\n                        $('#passes-list').append('<li>' + pass.name + ' - ' + pass.elevation_meters + ' meters (' + pass.location + ')<\/li>');\n                    });\n                },\n                error: function(error) {\n                    console.log(\"Error fetching data:\", error);\n                }\n            });\n        });\n    <\/script>\n<\/body>\n<\/html><\/code><\/pre>\n\n\n\n
      Explanation of the Sample Web Page:<\/h4>\n\n\n\n
        \n
      1. HTML Structure<\/strong>: The page contains a title, a button with the ID load-passes<\/code>, and an unordered list to display the mountain pass data.<\/li>\n\n\n\n
      2. AJAX Request<\/strong>: When the button is clicked, a jQuery AJAX request is made to the \/api\/mountain-passes<\/code> endpoint. If the request is successful, the data is displayed in the list. In case of an error, it\u2019s logged to the console.<\/li>\n<\/ol>\n\n\n\n
        To serve this web page, place the file in the static<\/code> directory of your Django project and configure your Django settings to serve static files.<\/p>\n\n\n\n
        Step 3: Setting Up Selenium with Firefox<\/strong><\/h4>\n\n\n\n
        Now, let\u2019s set up Selenium to automate the interaction with our local web page. We\u2019ll use Firefox<\/strong> with GeckoDriver<\/strong>, and the dotenv<\/strong> library will help us load environment variables for configuration.<\/p>\n\n\n\n
        Installing Dependencies<\/strong><\/h5>\n\n\n\n
        Make sure you have Selenium<\/strong>, dotenv<\/strong>, and GeckoDriver<\/strong> installed. Here\u2019s how to get started (I’m doing it on a Mac):<\/p>\n\n\n\n
        pip install selenium python-dotenv
        brew install geckodriver<\/code><\/pre>\n\n\n\n
        Step 4: Writing the Selenium Script<\/strong><\/h4>\n\n\n\n
        We\u2019ll write a Selenium script to automate the following tasks:<\/p>\n\n\n\n
          \n
        1. Navigate to the web page<\/strong>: http:\/\/127.0.0.1:8000\/mountain-passes\/<\/code><\/li>\n\n\n\n
        2. Find the “Load Passes” button<\/strong> and click it to trigger the AJAX request.<\/li>\n\n\n\n
        3. Capture the data<\/strong> returned from the \/api\/mountain-passes<\/code> endpoint.<\/li>\n\n\n\n
        4. Print and save the data<\/strong> in JSON format.<\/li>\n<\/ol>\n\n\n\n
          Here\u2019s the complete script:<\/p>\n\n\n\n
          import os\nimport time\nimport json\nfrom dotenv import load_dotenv\nfrom selenium import webdriver\nfrom selenium.webdriver.firefox.service import Service as FirefoxService\nfrom selenium.webdriver.common.by import By\nfrom selenium.webdriver.support.ui import WebDriverWait\nfrom selenium.webdriver.support import expected_conditions as EC\n\n# Load environment variables from the .env file\nload_dotenv()\n\n# Access the paths from the environment variables\nGECKO_DRIVER_PATH = os.getenv('GECKO_DRIVER_PATH')\n\n# Set up Firefox WebDriver\nservice = FirefoxService(GECKO_DRIVER_PATH)\ndriver = webdriver.Firefox(service=service)\n\n# Navigate to the web page\nprint(\"Navigating to the web page...\")\ndriver.get(\"http:\/\/127.0.0.1:8000\/mountain-passes\/\")\n\n# Find the load button and click it\ntry:\n    load_button = WebDriverWait(driver, 10).until(\n        EC.element_to_be_clickable((By.ID, \"load-passes\"))\n    )\n    print(\"Load button is visible and clickable.\")\n\n    # Scroll to the button using JavaScript\n    driver.execute_script(\"arguments[0].scrollIntoView();\", load_button)\n    print(\"Scrolled to the load button.\")\n\n    # Click the button using JavaScript\n    driver.execute_script(\"arguments[0].click();\", load_button)\n    print(\"Load button clicked via JavaScript.\")\nexcept Exception as e:\n    print(f\"Error finding or clicking the button: {e}\")\n\n# Add a sleep to allow the AJAX request to complete\ntime.sleep(5)\n\n# Execute JavaScript to capture the mountain passes data directly from the page\nmountain_passes_data = driver.execute_script(\"\"\"\n    return fetch('\/api\/mountain-passes')\n        .then(response => response.json())\n        .then(data => data);\n\"\"\")\n\n# Close the browser\ndriver.quit()\n\n# Print the captured data as JSON\nprint(\"Mountain Passes Data (JSON Format):\")\nprint(json.dumps(mountain_passes_data, indent=4))\n\n# Save the data to a JSON file\nwith open('mountain_passes_data.json', 'w') as json_file:\n    json.dump(mountain_passes_data, json_file, indent=4)\n\nprint(\"Mountain passes data has been saved to 'mountain_passes_data.json'.\")<\/code><\/pre>\n\n\n\n
          Step 5: Why This Approach?<\/strong> The Advantages of Scraping Dynamic APIs Over Traditional HTML Scraping<\/strong><\/h4>\n\n\n\n
          Scraping dynamic APIs is often more effective than parsing static HTML, especially for modern websites where content is loaded asynchronously. Here\u2019s why:<\/p>\n\n\n\n