Styjun

I need to validate a CSV file on the site CSVLint (https://csvlint.io)
Guide on using the API can be found here: https://csvlint.io/documentation

The problem is that when I request a JSON file from the site, I don't get proper response, I get an empty list. When I access the site from the browser using the authentication link, I get the full authentication results.
What am I doing wrong?

import requests


data = 
 'urls[]': 'https://www.wien.gv.at/statistik/ogd/vie_104.csv'

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
header='User-Agent' : user_agent 

response = requests.post('http://csvlint.io/package.json', data=data,headers=header)
first=response.json()
csvid=first['package']['url']+'.json'
#link='https'+csvid[5:]

request=requests.get(csvid).text
print(request)

The aim is to get the content of csvid JSON link.
In Python, in print(request) I get:

"version":"0.1","licence":"http://opendatacommons.org/licenses/odbl/","package":"validations":[]

In browser I get:

"version":"0.1","licence":"http://opendatacommons.org/licenses/odbl/","package":"validations":["url":"https://csvlint.io/validation/5c937a85b1b6fc0004000047","source":"https://www.wien.gv.at/statistik/ogd/vie_104.csv","state":"warnings"]

Try to simply delay your request like so:

import time
import requests


data = 
 'urls[]': 'https://www.wien.gv.at/statistik/ogd/vie_104.csv'

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
header='User-Agent' : user_agent 

response = requests.post('http://csvlint.io/package.json', data=data,headers=header)
first=response.json()
csvid=first['package']['url']+'.json'

time.sleep(5)

request=requests.get(csvid).text
print(request)

I get the correct output with this modification:

"version":"0.1","licence":"http://opendatacommons.org/licenses/odbl/","package":"validations":["url":"https://csvlint.io/validation/5c937a85b1b6fc0004000047","source":"https://www.wien.gv.at/statistik/ogd/vie_104.csv","state":"warnings"]

This is due to the way the package is handled with their API, as pointed out in another answer:

Note Currently the package is created in the background, so may not be
available immediately. This will change in the future.

There is a line in the API guidance that says:

Note Currently the package is created in the background, so may not be
available immediately. This will change in the future.

This is right at the stage where you are getting the empty return. Indeed, even with selenium a refresh is needed for info to appear.

It may be that you currently need a method like selenium for this step and possibly the final one. At the very least a delay for package to become ready.

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import json 

d = webdriver.Chrome()

data = 
 'urls[]': 'https://www.wien.gv.at/statistik/ogd/vie_104.csv'

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
header='User-Agent' : user_agent 

with requests.Session() as s:
 response = s.post('http://csvlint.io/package.json', data=data,headers=header)
 first = response.json()
 csvid = first['package']['url'] + '.json'
 d.get(csvid)
 d.refresh()
 data = WebDriverWait(d, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "pre"))).text 
 validationUrl = json.loads(data)['package']['validations'][0]['url'] 
 final = s.get(validationUrl + '.json').json()
 print(final)
 d.quit()