Styjun

I am trying to scrape a very simple table from a NOAA website: https://www1.ncdc.noaa.gov/pub/data/cdo/samples/PRECIP_HLY_sample_ascii.dat

The table is a ".dat" file and the site appears to be in html. When I use BeautifulSoup to read the content, I can see the content just fine. However, when I then search for the table with "find_all" or "find," I get nothing, i.e., [].

Here is my initial code:

page = requests.get('https://www1.ncdc.noaa.gov/pub/data/cdo/samples/PRECIP_HLY_sample_ascii.dat')
soup = BeautifulSoup(page.content,'html.parser') #'html5lib' #'html.parser' 'lxml'
table = soup.find_all('table') 

When I type soup, I get the following:

However, when I try to get the info into a table, it comes up blank

table
>> []

I have tried the following variations:

page = urllib.request.urlopen('https://www1.ncdc.noaa.gov/pub/data/cdo/samples/PRECIP_HLY_sample_ascii.dat').read()

soup = BeautifulSoup(page,'lxml')
soup = BeautifulSoup(page,'html5lib') #'' #''

table = soup.findAll('table') 
table = soup.findAll("div","class":"line-gutter-backdrop") 
table = soup.find_all(True)

However, table still comes up blank.

I found this question that appears to be similar:Cannot find table using Python BeautifulSoup

But my table is not in javascript (as far as I know). It is just text.

I am very new to data scraping and really have no idea why this simple example is not working. Any help is very appreciated. Thank you.

You cannot find the table tag because there is not, you have to find the pre tag.

You can try this snippet, it will get the text inside the table:

from bs4 import BeautifulSoup as soup
import urllib

url = 'https://www1.ncdc.noaa.gov/pub/data/cdo/samples/PRECIP_HLY_sample_ascii.dat'

response = urllib.request.urlopen(url)
html = response.read()

page_soup = soup(html,'lxml')
table = page_soup.find('p')
print(table.text)

OUTPUT:

STATION STATION_NAME ELEVATION LATITUDE LONGITUDE DATE HPCP Measurement Flag Quality Flag 
----------------- -------------------------------------------------- ---------- ---------- ---------- -------------- -------- ---------------- ------------ 
COOP:310301 ASHEVILLE NC US 682.1 35.5954 -82.5568 20100101 00:00 99999 ] 
COOP:310301 ASHEVILLE NC US 682.1 35.5954 -82.5568 20100101 01:00 0 g 
COOP:310301 ASHEVILLE NC US 682.1 35.5954 -82.5568 20100102 06:00 1