organizing data that I am pulling and saving to CSVWhy can't Python parse this JSON data?Save plot to image file instead of displaying it using MatplotlibHow do I write JSON data to a file?Pandas writing dataframe to CSV fileUnicode Encode error when writing to CSV after scrapingdictionary inside 2 loops to save to a single fileTrouble creating pandas dataframe from listsIteratively reading multiple cvs from different directories into dataframe then write to new csvcreating a live updating graph by using a pandas dataframe PythonWriting Web-Scrape Elements to csv file with desired formatting
Can compressed videos be decoded back to their uncompresed original format?
Can we compute the area of a quadrilateral with one right angle when we only know the lengths of any three sides?
Ambiguity in the definition of entropy
How do I know where to place holes on an instrument?
Avoiding direct proof while writing proof by induction
Am I breaking OOP practice with this architecture?
How to prevent "they're falling in love" trope
How can I prevent hyper evolved versions of regular creatures from wiping out their cousins?
Unlock My Phone! February 2018
How much of data wrangling is a data scientist's job?
Is it inappropriate for a student to attend their mentor's dissertation defense?
Can a virus destroy the BIOS of a modern computer?
How can I deal with my CEO asking me to hire someone with a higher salary than me, a co-founder?
What exploit are these user agents trying to use?
Why doesn't using multiple commands with a || or && conditional work?
How to show a landlord what we have in savings?
Intersection Puzzle
iPad being using in wall mount battery swollen
Why was the shrinking from 8″ made only to 5.25″ and not smaller (4″ or less)?
What reasons are there for a Capitalist to oppose a 100% inheritance tax?
Venezuelan girlfriend wants to travel the USA to be with me. What is the process?
What is the difference between 仮定 and 想定?
Little known, relatively unlikely, but scientifically plausible, apocalyptic (or near apocalyptic) events
Why can't we play rap on piano?
organizing data that I am pulling and saving to CSV
Why can't Python parse this JSON data?Save plot to image file instead of displaying it using MatplotlibHow do I write JSON data to a file?Pandas writing dataframe to CSV fileUnicode Encode error when writing to CSV after scrapingdictionary inside 2 loops to save to a single fileTrouble creating pandas dataframe from listsIteratively reading multiple cvs from different directories into dataframe then write to new csvcreating a live updating graph by using a pandas dataframe PythonWriting Web-Scrape Elements to csv file with desired formatting
-5
The Link below has a lot of layers to it's business listings. I am trying to pull all the contact info for each company. I have code that I used in the past for tables that I scraped in the past but it's a total mess and not pulling any results. Please help
https://www.estatesales.net/companies/NJ/Northern-New-Jersey
import pandas as pd
import csv
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
with open('file.csv','w') as csvfile
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
#Write the headers
writer.writerow(['Name','City','State','Telephone'])
#convert second column of 1st dataframe to a list and write to csv file
writer.writerow(list(dfs[0][1])[:-1])
```
python web-scraping scrapy screen-scraping export-to-csv
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
add a comment |
-5
The Link below has a lot of layers to it's business listings. I am trying to pull all the contact info for each company. I have code that I used in the past for tables that I scraped in the past but it's a total mess and not pulling any results. Please help
https://www.estatesales.net/companies/NJ/Northern-New-Jersey
import pandas as pd
import csv
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
with open('file.csv','w') as csvfile
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
#Write the headers
writer.writerow(['Name','City','State','Telephone'])
#convert second column of 1st dataframe to a list and write to csv file
writer.writerow(list(dfs[0][1])[:-1])
```
python web-scraping scrapy screen-scraping export-to-csv
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
What isdfsin your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.
– Luca Bezerra
Mar 21 at 23:45
add a comment |
-5
-5
-5
The Link below has a lot of layers to it's business listings. I am trying to pull all the contact info for each company. I have code that I used in the past for tables that I scraped in the past but it's a total mess and not pulling any results. Please help
https://www.estatesales.net/companies/NJ/Northern-New-Jersey
import pandas as pd
import csv
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
with open('file.csv','w') as csvfile
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
#Write the headers
writer.writerow(['Name','City','State','Telephone'])
#convert second column of 1st dataframe to a list and write to csv file
writer.writerow(list(dfs[0][1])[:-1])
```
python web-scraping scrapy screen-scraping export-to-csv
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
The Link below has a lot of layers to it's business listings. I am trying to pull all the contact info for each company. I have code that I used in the past for tables that I scraped in the past but it's a total mess and not pulling any results. Please help
https://www.estatesales.net/companies/NJ/Northern-New-Jersey
import pandas as pd
import csv
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
with open('file.csv','w') as csvfile
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
#Write the headers
writer.writerow(['Name','City','State','Telephone'])
#convert second column of 1st dataframe to a list and write to csv file
writer.writerow(list(dfs[0][1])[:-1])
```
python web-scraping scrapy screen-scraping export-to-csv
python web-scraping scrapy screen-scraping export-to-csv
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
edited Mar 21 at 22:56
Michael Gordon
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
asked Mar 21 at 21:28
Michael GordonMichael Gordon
33
33
What isdfsin your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.
– Luca Bezerra
Mar 21 at 23:45
add a comment |
What isdfsin your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.
– Luca Bezerra
Mar 21 at 23:45
What is
dfs in your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.– Luca Bezerra
Mar 21 at 23:45
What is
dfs in your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.– Luca Bezerra
Mar 21 at 23:45
add a comment |
1 Answer
1
active
oldest
votes
0
You can use pandas to do that. Collect all the data into a dataframe, then just write the dataframe to file.
import pandas as pd
import requests
import bs4
root_url = 'https://www.estatesales.net'
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
results = pd.DataFrame()
for url in url_list:
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
companies = soup.find_all('app-company-city-view-row')
for company in companies:
try:
link = root_url + company.find('a','itemprop':"name url")['href']
print(link)
except:
continue
response = requests.get(link)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
company_info = soup.find('div', 'id':'CompanyInfo')
try:
name = company_info.find('h1','itemprop':"name").text
except:
name = 'N/A'
try:
city = company_info.find('span','itemprop':"addressLocality").text
except:
city = 'N/A'
try:
state = company_info.find('span','itemprop':"addressRegion").text
except:
state = 'N/A'
try:
phone = company_info.find('span','itemprop':"telephone").text
except:
phone = 'N/A'
temp_df = pd.DataFrame([[name, city, state, phone]], columns = ['Name','City','State','Telephone'])
results = results.append(temp_df).reset_index(drop=True)
results.to_csv('file.csv', index=False)
Output:
print (results)
Name ... Telephone
0 Hub Estate Liquidation ... (862) 259-5364
1 Pink Dog Estate And Moving Sales, LLC ... (201) 674-7464
2 Remember When Antiques And Estate Sales, LLC ... (917) 410-7100
3 Always Nostalgia ... (201) 388-2598
4 Before & After Corp ... (201) 747-5342
5 Discovery Estate Sales ... (908) 620-1776
6 Plum Cottage Estate Sales & Appraisals ... (732) 788-4101
7 Decorate On A Dime, LLC ... (908) 380-3340
8 Remmey Antiques & Fine Art Appraisers & Auctio... ... (973) 425-1608
9 Easy Picking Estate Sales ... (917) 691-6132
10 EstateSalesByOlga.com ... (908) 337-4240
11 Real McCoy ... (973) 418-1286
12 Then And Now Estate Sales ... (201) 259-8408
13 Insideout,llc ... (215) 630-4942
14 Beacon Hill Estate Sales & Appraisals LLC ... (908) 601-5381
15 Lori Palmer Estate Sales ... (732) 809-3382
16 Somerset Appraisal And Estate Services ... (908) 872-6236
17 Central Jersey Estate Sales & Liquidators ... (908) 625-1622
18 C. T. Peters Inc., Appraisers ... (732) 747-9450
19 Treasures Of Yesterday Estate Sales ... (201) 446-2790
20 Caring Transitions Of Central Jersey ... (732) 307-3881
21 Attic To The Basement Estate Sales ... (732) 778-7674
22 Griffin Estate Sales ... (908) 447-3044
23 Dodge Estate Sales LLC ... (973) 714-1401
24 Estate Sales By Kathy ... (732) 674-7330
25 Curated Estates ... (917) 470-9255
26 Brownstone Liquidators ... (845) 821-3254
27 Jersey Estate Sales ... (973) 428-1906
28 Arkay Resale, Inc. ... (201) 741-4165
29 Vinylpiggy's LLC ... (551) 804-7152
.. ... ... ...
116 Classic Estate Sales & Appraisals ... (201) 370-4021
117 Sullivintage ... (732) 890-3485
118 A Trotters Sale ... (973) 819-8685
119 Shore Estate Sales ... (732) 616-3371
120 Liberty Antiques ... (908) 581-6987
121 Johnson Estate Sales ... (201) 259-0442
122 Buy-Gone Trading Co. ... (201) 665-8208
123 Elite Auctions & Estate Sales ... (732) 751-1112
124 The Butler Did It ... (908) 892-8133
125 Drama Mama Home Estate Sales ... (862) 400-2081
126 New Jersey Gold Resources ... N/A
127 Lyrix Inc ... (973) 632-1600
128 Home Ready Services, LLC ... (908) 370-3062
129 Bygone Days Estate Sales, LLC ... (973) 857-9069
130 Fourty Fifty Sixty ... (973) 341-7891
131 True Salvage ... (973) 970-5400
132 Ina's Antiques & Estate Sales ... (908) 578-4118
133 Caring Transitions Jersey Shore ... (732) 681-0054
134 ENCORE ESTATE & TAG SALES ... (973) 220-4611
135 Ajtrains ... (732) 859-1606
136 M.T. House Estate & Moving Sales ... (973) 865-1173
137 WCL Antiques ... (201) 739-3173
138 Kens Antiques ... (732) 306-2717
139 Lee Dowdy Antiques ... (201) 650-7208
140 Garage Sale Goddesses ... (201) 612-8510
141 M & J Estate Sales ... (908) 956-4284
142 Granny's Attic ... (201) 632-0102
143 Red Barn Estate sales ... (201) 481-5428
144 D&r Estate & Tag Sales ... (201) 573-1009
145 Always Remember When LLC ... (347) 244-1591
[146 rows x 4 columns]
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55289520%2forganizing-data-that-i-am-pulling-and-saving-to-csv%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
You can use pandas to do that. Collect all the data into a dataframe, then just write the dataframe to file.
import pandas as pd
import requests
import bs4
root_url = 'https://www.estatesales.net'
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
results = pd.DataFrame()
for url in url_list:
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
companies = soup.find_all('app-company-city-view-row')
for company in companies:
try:
link = root_url + company.find('a','itemprop':"name url")['href']
print(link)
except:
continue
response = requests.get(link)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
company_info = soup.find('div', 'id':'CompanyInfo')
try:
name = company_info.find('h1','itemprop':"name").text
except:
name = 'N/A'
try:
city = company_info.find('span','itemprop':"addressLocality").text
except:
city = 'N/A'
try:
state = company_info.find('span','itemprop':"addressRegion").text
except:
state = 'N/A'
try:
phone = company_info.find('span','itemprop':"telephone").text
except:
phone = 'N/A'
temp_df = pd.DataFrame([[name, city, state, phone]], columns = ['Name','City','State','Telephone'])
results = results.append(temp_df).reset_index(drop=True)
results.to_csv('file.csv', index=False)
Output:
print (results)
Name ... Telephone
0 Hub Estate Liquidation ... (862) 259-5364
1 Pink Dog Estate And Moving Sales, LLC ... (201) 674-7464
2 Remember When Antiques And Estate Sales, LLC ... (917) 410-7100
3 Always Nostalgia ... (201) 388-2598
4 Before & After Corp ... (201) 747-5342
5 Discovery Estate Sales ... (908) 620-1776
6 Plum Cottage Estate Sales & Appraisals ... (732) 788-4101
7 Decorate On A Dime, LLC ... (908) 380-3340
8 Remmey Antiques & Fine Art Appraisers & Auctio... ... (973) 425-1608
9 Easy Picking Estate Sales ... (917) 691-6132
10 EstateSalesByOlga.com ... (908) 337-4240
11 Real McCoy ... (973) 418-1286
12 Then And Now Estate Sales ... (201) 259-8408
13 Insideout,llc ... (215) 630-4942
14 Beacon Hill Estate Sales & Appraisals LLC ... (908) 601-5381
15 Lori Palmer Estate Sales ... (732) 809-3382
16 Somerset Appraisal And Estate Services ... (908) 872-6236
17 Central Jersey Estate Sales & Liquidators ... (908) 625-1622
18 C. T. Peters Inc., Appraisers ... (732) 747-9450
19 Treasures Of Yesterday Estate Sales ... (201) 446-2790
20 Caring Transitions Of Central Jersey ... (732) 307-3881
21 Attic To The Basement Estate Sales ... (732) 778-7674
22 Griffin Estate Sales ... (908) 447-3044
23 Dodge Estate Sales LLC ... (973) 714-1401
24 Estate Sales By Kathy ... (732) 674-7330
25 Curated Estates ... (917) 470-9255
26 Brownstone Liquidators ... (845) 821-3254
27 Jersey Estate Sales ... (973) 428-1906
28 Arkay Resale, Inc. ... (201) 741-4165
29 Vinylpiggy's LLC ... (551) 804-7152
.. ... ... ...
116 Classic Estate Sales & Appraisals ... (201) 370-4021
117 Sullivintage ... (732) 890-3485
118 A Trotters Sale ... (973) 819-8685
119 Shore Estate Sales ... (732) 616-3371
120 Liberty Antiques ... (908) 581-6987
121 Johnson Estate Sales ... (201) 259-0442
122 Buy-Gone Trading Co. ... (201) 665-8208
123 Elite Auctions & Estate Sales ... (732) 751-1112
124 The Butler Did It ... (908) 892-8133
125 Drama Mama Home Estate Sales ... (862) 400-2081
126 New Jersey Gold Resources ... N/A
127 Lyrix Inc ... (973) 632-1600
128 Home Ready Services, LLC ... (908) 370-3062
129 Bygone Days Estate Sales, LLC ... (973) 857-9069
130 Fourty Fifty Sixty ... (973) 341-7891
131 True Salvage ... (973) 970-5400
132 Ina's Antiques & Estate Sales ... (908) 578-4118
133 Caring Transitions Jersey Shore ... (732) 681-0054
134 ENCORE ESTATE & TAG SALES ... (973) 220-4611
135 Ajtrains ... (732) 859-1606
136 M.T. House Estate & Moving Sales ... (973) 865-1173
137 WCL Antiques ... (201) 739-3173
138 Kens Antiques ... (732) 306-2717
139 Lee Dowdy Antiques ... (201) 650-7208
140 Garage Sale Goddesses ... (201) 612-8510
141 M & J Estate Sales ... (908) 956-4284
142 Granny's Attic ... (201) 632-0102
143 Red Barn Estate sales ... (201) 481-5428
144 D&r Estate & Tag Sales ... (201) 573-1009
145 Always Remember When LLC ... (347) 244-1591
[146 rows x 4 columns]
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
add a comment |
0
You can use pandas to do that. Collect all the data into a dataframe, then just write the dataframe to file.
import pandas as pd
import requests
import bs4
root_url = 'https://www.estatesales.net'
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
results = pd.DataFrame()
for url in url_list:
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
companies = soup.find_all('app-company-city-view-row')
for company in companies:
try:
link = root_url + company.find('a','itemprop':"name url")['href']
print(link)
except:
continue
response = requests.get(link)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
company_info = soup.find('div', 'id':'CompanyInfo')
try:
name = company_info.find('h1','itemprop':"name").text
except:
name = 'N/A'
try:
city = company_info.find('span','itemprop':"addressLocality").text
except:
city = 'N/A'
try:
state = company_info.find('span','itemprop':"addressRegion").text
except:
state = 'N/A'
try:
phone = company_info.find('span','itemprop':"telephone").text
except:
phone = 'N/A'
temp_df = pd.DataFrame([[name, city, state, phone]], columns = ['Name','City','State','Telephone'])
results = results.append(temp_df).reset_index(drop=True)
results.to_csv('file.csv', index=False)
Output:
print (results)
Name ... Telephone
0 Hub Estate Liquidation ... (862) 259-5364
1 Pink Dog Estate And Moving Sales, LLC ... (201) 674-7464
2 Remember When Antiques And Estate Sales, LLC ... (917) 410-7100
3 Always Nostalgia ... (201) 388-2598
4 Before & After Corp ... (201) 747-5342
5 Discovery Estate Sales ... (908) 620-1776
6 Plum Cottage Estate Sales & Appraisals ... (732) 788-4101
7 Decorate On A Dime, LLC ... (908) 380-3340
8 Remmey Antiques & Fine Art Appraisers & Auctio... ... (973) 425-1608
9 Easy Picking Estate Sales ... (917) 691-6132
10 EstateSalesByOlga.com ... (908) 337-4240
11 Real McCoy ... (973) 418-1286
12 Then And Now Estate Sales ... (201) 259-8408
13 Insideout,llc ... (215) 630-4942
14 Beacon Hill Estate Sales & Appraisals LLC ... (908) 601-5381
15 Lori Palmer Estate Sales ... (732) 809-3382
16 Somerset Appraisal And Estate Services ... (908) 872-6236
17 Central Jersey Estate Sales & Liquidators ... (908) 625-1622
18 C. T. Peters Inc., Appraisers ... (732) 747-9450
19 Treasures Of Yesterday Estate Sales ... (201) 446-2790
20 Caring Transitions Of Central Jersey ... (732) 307-3881
21 Attic To The Basement Estate Sales ... (732) 778-7674
22 Griffin Estate Sales ... (908) 447-3044
23 Dodge Estate Sales LLC ... (973) 714-1401
24 Estate Sales By Kathy ... (732) 674-7330
25 Curated Estates ... (917) 470-9255
26 Brownstone Liquidators ... (845) 821-3254
27 Jersey Estate Sales ... (973) 428-1906
28 Arkay Resale, Inc. ... (201) 741-4165
29 Vinylpiggy's LLC ... (551) 804-7152
.. ... ... ...
116 Classic Estate Sales & Appraisals ... (201) 370-4021
117 Sullivintage ... (732) 890-3485
118 A Trotters Sale ... (973) 819-8685
119 Shore Estate Sales ... (732) 616-3371
120 Liberty Antiques ... (908) 581-6987
121 Johnson Estate Sales ... (201) 259-0442
122 Buy-Gone Trading Co. ... (201) 665-8208
123 Elite Auctions & Estate Sales ... (732) 751-1112
124 The Butler Did It ... (908) 892-8133
125 Drama Mama Home Estate Sales ... (862) 400-2081
126 New Jersey Gold Resources ... N/A
127 Lyrix Inc ... (973) 632-1600
128 Home Ready Services, LLC ... (908) 370-3062
129 Bygone Days Estate Sales, LLC ... (973) 857-9069
130 Fourty Fifty Sixty ... (973) 341-7891
131 True Salvage ... (973) 970-5400
132 Ina's Antiques & Estate Sales ... (908) 578-4118
133 Caring Transitions Jersey Shore ... (732) 681-0054
134 ENCORE ESTATE & TAG SALES ... (973) 220-4611
135 Ajtrains ... (732) 859-1606
136 M.T. House Estate & Moving Sales ... (973) 865-1173
137 WCL Antiques ... (201) 739-3173
138 Kens Antiques ... (732) 306-2717
139 Lee Dowdy Antiques ... (201) 650-7208
140 Garage Sale Goddesses ... (201) 612-8510
141 M & J Estate Sales ... (908) 956-4284
142 Granny's Attic ... (201) 632-0102
143 Red Barn Estate sales ... (201) 481-5428
144 D&r Estate & Tag Sales ... (201) 573-1009
145 Always Remember When LLC ... (347) 244-1591
[146 rows x 4 columns]
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
add a comment |
0
0
0
You can use pandas to do that. Collect all the data into a dataframe, then just write the dataframe to file.
import pandas as pd
import requests
import bs4
root_url = 'https://www.estatesales.net'
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
results = pd.DataFrame()
for url in url_list:
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
companies = soup.find_all('app-company-city-view-row')
for company in companies:
try:
link = root_url + company.find('a','itemprop':"name url")['href']
print(link)
except:
continue
response = requests.get(link)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
company_info = soup.find('div', 'id':'CompanyInfo')
try:
name = company_info.find('h1','itemprop':"name").text
except:
name = 'N/A'
try:
city = company_info.find('span','itemprop':"addressLocality").text
except:
city = 'N/A'
try:
state = company_info.find('span','itemprop':"addressRegion").text
except:
state = 'N/A'
try:
phone = company_info.find('span','itemprop':"telephone").text
except:
phone = 'N/A'
temp_df = pd.DataFrame([[name, city, state, phone]], columns = ['Name','City','State','Telephone'])
results = results.append(temp_df).reset_index(drop=True)
results.to_csv('file.csv', index=False)
Output:
print (results)
Name ... Telephone
0 Hub Estate Liquidation ... (862) 259-5364
1 Pink Dog Estate And Moving Sales, LLC ... (201) 674-7464
2 Remember When Antiques And Estate Sales, LLC ... (917) 410-7100
3 Always Nostalgia ... (201) 388-2598
4 Before & After Corp ... (201) 747-5342
5 Discovery Estate Sales ... (908) 620-1776
6 Plum Cottage Estate Sales & Appraisals ... (732) 788-4101
7 Decorate On A Dime, LLC ... (908) 380-3340
8 Remmey Antiques & Fine Art Appraisers & Auctio... ... (973) 425-1608
9 Easy Picking Estate Sales ... (917) 691-6132
10 EstateSalesByOlga.com ... (908) 337-4240
11 Real McCoy ... (973) 418-1286
12 Then And Now Estate Sales ... (201) 259-8408
13 Insideout,llc ... (215) 630-4942
14 Beacon Hill Estate Sales & Appraisals LLC ... (908) 601-5381
15 Lori Palmer Estate Sales ... (732) 809-3382
16 Somerset Appraisal And Estate Services ... (908) 872-6236
17 Central Jersey Estate Sales & Liquidators ... (908) 625-1622
18 C. T. Peters Inc., Appraisers ... (732) 747-9450
19 Treasures Of Yesterday Estate Sales ... (201) 446-2790
20 Caring Transitions Of Central Jersey ... (732) 307-3881
21 Attic To The Basement Estate Sales ... (732) 778-7674
22 Griffin Estate Sales ... (908) 447-3044
23 Dodge Estate Sales LLC ... (973) 714-1401
24 Estate Sales By Kathy ... (732) 674-7330
25 Curated Estates ... (917) 470-9255
26 Brownstone Liquidators ... (845) 821-3254
27 Jersey Estate Sales ... (973) 428-1906
28 Arkay Resale, Inc. ... (201) 741-4165
29 Vinylpiggy's LLC ... (551) 804-7152
.. ... ... ...
116 Classic Estate Sales & Appraisals ... (201) 370-4021
117 Sullivintage ... (732) 890-3485
118 A Trotters Sale ... (973) 819-8685
119 Shore Estate Sales ... (732) 616-3371
120 Liberty Antiques ... (908) 581-6987
121 Johnson Estate Sales ... (201) 259-0442
122 Buy-Gone Trading Co. ... (201) 665-8208
123 Elite Auctions & Estate Sales ... (732) 751-1112
124 The Butler Did It ... (908) 892-8133
125 Drama Mama Home Estate Sales ... (862) 400-2081
126 New Jersey Gold Resources ... N/A
127 Lyrix Inc ... (973) 632-1600
128 Home Ready Services, LLC ... (908) 370-3062
129 Bygone Days Estate Sales, LLC ... (973) 857-9069
130 Fourty Fifty Sixty ... (973) 341-7891
131 True Salvage ... (973) 970-5400
132 Ina's Antiques & Estate Sales ... (908) 578-4118
133 Caring Transitions Jersey Shore ... (732) 681-0054
134 ENCORE ESTATE & TAG SALES ... (973) 220-4611
135 Ajtrains ... (732) 859-1606
136 M.T. House Estate & Moving Sales ... (973) 865-1173
137 WCL Antiques ... (201) 739-3173
138 Kens Antiques ... (732) 306-2717
139 Lee Dowdy Antiques ... (201) 650-7208
140 Garage Sale Goddesses ... (201) 612-8510
141 M & J Estate Sales ... (908) 956-4284
142 Granny's Attic ... (201) 632-0102
143 Red Barn Estate sales ... (201) 481-5428
144 D&r Estate & Tag Sales ... (201) 573-1009
145 Always Remember When LLC ... (347) 244-1591
[146 rows x 4 columns]
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
You can use pandas to do that. Collect all the data into a dataframe, then just write the dataframe to file.
import pandas as pd
import requests
import bs4
root_url = 'https://www.estatesales.net'
url_list=['https://www.estatesales.net/companies/NJ/Northern-New-Jersey']
results = pd.DataFrame()
for url in url_list:
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
companies = soup.find_all('app-company-city-view-row')
for company in companies:
try:
link = root_url + company.find('a','itemprop':"name url")['href']
print(link)
except:
continue
response = requests.get(link)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
company_info = soup.find('div', 'id':'CompanyInfo')
try:
name = company_info.find('h1','itemprop':"name").text
except:
name = 'N/A'
try:
city = company_info.find('span','itemprop':"addressLocality").text
except:
city = 'N/A'
try:
state = company_info.find('span','itemprop':"addressRegion").text
except:
state = 'N/A'
try:
phone = company_info.find('span','itemprop':"telephone").text
except:
phone = 'N/A'
temp_df = pd.DataFrame([[name, city, state, phone]], columns = ['Name','City','State','Telephone'])
results = results.append(temp_df).reset_index(drop=True)
results.to_csv('file.csv', index=False)
Output:
print (results)
Name ... Telephone
0 Hub Estate Liquidation ... (862) 259-5364
1 Pink Dog Estate And Moving Sales, LLC ... (201) 674-7464
2 Remember When Antiques And Estate Sales, LLC ... (917) 410-7100
3 Always Nostalgia ... (201) 388-2598
4 Before & After Corp ... (201) 747-5342
5 Discovery Estate Sales ... (908) 620-1776
6 Plum Cottage Estate Sales & Appraisals ... (732) 788-4101
7 Decorate On A Dime, LLC ... (908) 380-3340
8 Remmey Antiques & Fine Art Appraisers & Auctio... ... (973) 425-1608
9 Easy Picking Estate Sales ... (917) 691-6132
10 EstateSalesByOlga.com ... (908) 337-4240
11 Real McCoy ... (973) 418-1286
12 Then And Now Estate Sales ... (201) 259-8408
13 Insideout,llc ... (215) 630-4942
14 Beacon Hill Estate Sales & Appraisals LLC ... (908) 601-5381
15 Lori Palmer Estate Sales ... (732) 809-3382
16 Somerset Appraisal And Estate Services ... (908) 872-6236
17 Central Jersey Estate Sales & Liquidators ... (908) 625-1622
18 C. T. Peters Inc., Appraisers ... (732) 747-9450
19 Treasures Of Yesterday Estate Sales ... (201) 446-2790
20 Caring Transitions Of Central Jersey ... (732) 307-3881
21 Attic To The Basement Estate Sales ... (732) 778-7674
22 Griffin Estate Sales ... (908) 447-3044
23 Dodge Estate Sales LLC ... (973) 714-1401
24 Estate Sales By Kathy ... (732) 674-7330
25 Curated Estates ... (917) 470-9255
26 Brownstone Liquidators ... (845) 821-3254
27 Jersey Estate Sales ... (973) 428-1906
28 Arkay Resale, Inc. ... (201) 741-4165
29 Vinylpiggy's LLC ... (551) 804-7152
.. ... ... ...
116 Classic Estate Sales & Appraisals ... (201) 370-4021
117 Sullivintage ... (732) 890-3485
118 A Trotters Sale ... (973) 819-8685
119 Shore Estate Sales ... (732) 616-3371
120 Liberty Antiques ... (908) 581-6987
121 Johnson Estate Sales ... (201) 259-0442
122 Buy-Gone Trading Co. ... (201) 665-8208
123 Elite Auctions & Estate Sales ... (732) 751-1112
124 The Butler Did It ... (908) 892-8133
125 Drama Mama Home Estate Sales ... (862) 400-2081
126 New Jersey Gold Resources ... N/A
127 Lyrix Inc ... (973) 632-1600
128 Home Ready Services, LLC ... (908) 370-3062
129 Bygone Days Estate Sales, LLC ... (973) 857-9069
130 Fourty Fifty Sixty ... (973) 341-7891
131 True Salvage ... (973) 970-5400
132 Ina's Antiques & Estate Sales ... (908) 578-4118
133 Caring Transitions Jersey Shore ... (732) 681-0054
134 ENCORE ESTATE & TAG SALES ... (973) 220-4611
135 Ajtrains ... (732) 859-1606
136 M.T. House Estate & Moving Sales ... (973) 865-1173
137 WCL Antiques ... (201) 739-3173
138 Kens Antiques ... (732) 306-2717
139 Lee Dowdy Antiques ... (201) 650-7208
140 Garage Sale Goddesses ... (201) 612-8510
141 M & J Estate Sales ... (908) 956-4284
142 Granny's Attic ... (201) 632-0102
143 Red Barn Estate sales ... (201) 481-5428
144 D&r Estate & Tag Sales ... (201) 573-1009
145 Always Remember When LLC ... (347) 244-1591
[146 rows x 4 columns]
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
answered Mar 22 at 0:45
chitown88chitown88
5,6431627
5,6431627
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55289520%2forganizing-data-that-i-am-pulling-and-saving-to-csv%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What is
dfsin your code? You're never actually fetching the url in order to get its contents, it seems like there are some parts of your code missing.– Luca Bezerra
Mar 21 at 23:45