Styjun

I'm trying to scrape data from Google Patents and finding the execution time takes too long. How can I increase the speed? Running through 8000 patents took 7 hours already...

Here's an example of a patent.

I need to get data from the tables below and write them to a csv file. I think the bottle neck is at WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[@class='table style-scope patent-result']")))

Is this necessary or can I use find_elements_by_css_selector and check if that returns anything?

#...
from selenium.webdriver.support import expected_conditions as EC
#...

## read file of patent numbers and initiate chrome

url = "https://patents.google.com/patent/US6403086B1/en?oq=US6403086B1"

for x in patent_number:

 #url = new url with new patent number similar to above

 try: 
 driver.get(url) 
 driver.set_page_load_timeout(20) 
 except: 
 #--write to csv
 continue

 if "404" in driver.title: #patent number not found
 #--write to csv
 continue

 try: 
 WebDriverWait(driver, 10).until(
 EC.presence_of_element_located((By.XPATH, "//div[@class='table style-scope patent-result']"))
 )
 except: 
 #--write to csv
 continue


 ## rest of code to get data from tables and write to csv

Is there a more efficient way of finding if these tables exist on a patent page? Or would there be a difference if I used BeautifulSoup?

I'm new to webscraping, so any help would be greatly appreciated :)

Not sure which tables you are after but consider you may be able to use requests and pandas to grab the tables, as well as Session to re-use connection.

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

codes = ['US6403086B1','US6403086B1'] #patent numbers to come from file
with requests.Session() as s:
 for code in codes:
 url = 'https://patents.google.com/patent//en?oq='.format(code, code)
 r = s.get(url)
 tables = pd.read_html(str(r.content))
 print(tables) #example only. Remove later
 #here would add some tidying up to tables e.g. dropNa rows, replace NaN with '' .... 
 # rather than print... whatever steps to store info you want until write out