Styjun

Would it be appropriate to sand a floor between coats of poly with a handheld orbital sander?

A horrible Stockfish chess engine evaluation

Why did people still chant "Lock her up" at Trump rallies in 2019?

Efficiently defining a SparseArray function

Chorophyll and photosynthesis in plants with coloured leaves

What happens when adult Billy Batson says "Shazam"?

What are the indigenous English words for a prostitute?

What is the parallel of Day of the Dead with Stranger things?

Why do we need common sense in AI?

Graduate student with abysmal English writing skills, how to help

When did "&" stop being taught alongside the alphabet?

How to know if blackberries are safe to eat

Intelligent Ants in the Amazon

Swapping "Good" and "Bad"

How can I effectively communicate to recruiters that a phone call is not possible?

How do we handle pauses in a dialogue?

Why does wrapping Aluminium foil around my food help it keep warm, aluminium be good conductor should have no effect?

Credit score and financing new car

Integer Lists of Noah

What is this little owl-like bird?

How effective would wooden scale armor be in a medieval setting?

What is the right approach to quit a job during probation period for a competing offer?

Reverse dots and boxes

Having decision making power over someone's assets

use node js to do web crawler but can not search data

How to append to a file in Node?Using node.js as a simple web serverHow can I update NodeJS and NPM to the next versions?How to uninstall npm modules in node js?How to show data from mysql in nodejs with a refresh ratenode how to create a directory if doesn't exist?Nodes http request before an async.mapformidable parse callback not be called and no errorWait for data from external API before making POST requestNode JS Promise TypeError: Cannot read property 'then' of undefined

const request = require('request')
const ironmans = [
 'https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm? 
 type=Weather&time=7Day',
]

async.map( ironmans, getInfo, (err, results)=>
console.log(results);
)

function getInfo(url, callback)
request(url, function(err, res, body)
var $ = cheerio.load(body)
const result = []; 
const table_tr = $(".Forecast-box table tr"); 
for (let i = 1; i <3; i++) // tr
 const table_td = table_tr.eq(i).find('td'); // (td)
 const time = table_td.eq(2).text(); // time
 const temperture = table_td.eq(4).text(); // temperture 
 const weather = table_td.eq(4).text(); // weather 
 result.push(Object.assign( time, temperture, weather ));
 
 console.log(result);
 )

const request = require('request')
const ironmans = [
 'https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm? 
 type=Weather&time=7Day',
]

async.map( ironmans, getInfo, (err, results)=>
console.log(results);
)

function getInfo(url, callback)
request(url, function(err, res, body)
var $ = cheerio.load(body)
const result = []; 
const table_tr = $(".Forecast-box table tr"); 
for (let i = 1; i <3; i++) // tr
 const table_td = table_tr.eq(i).find('td'); // (td)
 const time = table_td.eq(2).text(); // time
 const temperture = table_td.eq(4).text(); // temperture 
 const weather = table_td.eq(4).text(); // weather 
 result.push(Object.assign( time, temperture, weather ));
 
 console.log(result);
 )

const request = require('request')
const ironmans = [
 'https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm? 
 type=Weather&time=7Day',
]

async.map( ironmans, getInfo, (err, results)=>
console.log(results);
)

function getInfo(url, callback)
request(url, function(err, res, body)
var $ = cheerio.load(body)
const result = []; 
const table_tr = $(".Forecast-box table tr"); 
for (let i = 1; i <3; i++) // tr
 const table_td = table_tr.eq(i).find('td'); // (td)
 const time = table_td.eq(2).text(); // time
 const temperture = table_td.eq(4).text(); // temperture 
 const weather = table_td.eq(4).text(); // weather 
 result.push(Object.assign( time, temperture, weather ));
 
 console.log(result);
 )

const request = require('request')
const ironmans = [
 'https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm? 
 type=Weather&time=7Day',
]

async.map( ironmans, getInfo, (err, results)=>
console.log(results);
)

function getInfo(url, callback)
request(url, function(err, res, body)
var $ = cheerio.load(body)
const result = []; 
const table_tr = $(".Forecast-box table tr"); 
for (let i = 1; i <3; i++) // tr
 const table_td = table_tr.eq(i).find('td'); // (td)
 const time = table_td.eq(2).text(); // time
 const temperture = table_td.eq(4).text(); // temperture 
 const weather = table_td.eq(4).text(); // weather 
 result.push(Object.assign( time, temperture, weather ));
 
 console.log(result);
 )

1 Answer
1

active

oldest

votes

0

The weather table has been appended to the page by a xhr request, that mean if you use request to get the page resource, then you can not get the data table(like you View page source). Try another ways (headless browser), simple way crawl https://www.cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm instead of https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm?
type=Weather&time=7Day

share|improve this answer

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  how I can know to use this website? where can I find? cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm
  
  – user8449442
  Mar 26 at 1:24
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @samuraikyo37 Open developer tool (Chrome browser), in Network tab, when you click to １週預報, 過去２４小時 or 逐３小時預報 button, a network request has been fired. This is what you want.
  
  – hoangdv
  Mar 26 at 1:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  hoangdv I know thanks.
  
  – user8449442
  Mar 26 at 2:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I am wondering that why I cant get weather (which column have sun and moon picture), I try : weather =table_tr.eq(3).text(); / or const a = $(".Forecast-box table tr td title"); weather = a.eq(1).text(); still not work. how I can do? thanks
  
  – user8449442
  Mar 27 at 8:35
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55348369%2fuse-node-js-to-do-web-crawler-but-can-not-search-data%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

The weather table has been appended to the page by a xhr request, that mean if you use request to get the page resource, then you can not get the data table(like you View page source). Try another ways (headless browser), simple way crawl https://www.cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm instead of https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm?
type=Weather&time=7Day

share|improve this answer

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  how I can know to use this website? where can I find? cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm
  
  – user8449442
  Mar 26 at 1:24
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @samuraikyo37 Open developer tool (Chrome browser), in Network tab, when you click to １週預報, 過去２４小時 or 逐３小時預報 button, a network request has been fired. This is what you want.
  
  – hoangdv
  Mar 26 at 1:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  hoangdv I know thanks.
  
  – user8449442
  Mar 26 at 2:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I am wondering that why I cant get weather (which column have sun and moon picture), I try : weather =table_tr.eq(3).text(); / or const a = $(".Forecast-box table tr td title"); weather = a.eq(1).text(); still not work. how I can do? thanks
  
  – user8449442
  Mar 27 at 8:35
  

  
  
  
  

add a comment | 

0

The weather table has been appended to the page by a xhr request, that mean if you use request to get the page resource, then you can not get the data table(like you View page source). Try another ways (headless browser), simple way crawl https://www.cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm instead of https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm?
type=Weather&time=7Day

share|improve this answer

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  how I can know to use this website? where can I find? cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm
  
  – user8449442
  Mar 26 at 1:24
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @samuraikyo37 Open developer tool (Chrome browser), in Network tab, when you click to １週預報, 過去２４小時 or 逐３小時預報 button, a network request has been fired. This is what you want.
  
  – hoangdv
  Mar 26 at 1:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  hoangdv I know thanks.
  
  – user8449442
  Mar 26 at 2:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I am wondering that why I cant get weather (which column have sun and moon picture), I try : weather =table_tr.eq(3).text(); / or const a = $(".Forecast-box table tr td title"); weather = a.eq(1).text(); still not work. how I can do? thanks
  
  – user8449442
  Mar 27 at 8:35
  

  
  
  
  

add a comment | 

The weather table has been appended to the page by a xhr request, that mean if you use request to get the page resource, then you can not get the data table(like you View page source). Try another ways (headless browser), simple way crawl https://www.cwb.gov.tw/V7/forecast/town368/3Hr/6400800.htm instead of https://www.cwb.gov.tw/V7/forecast/town368/towns/6400800.htm?
type=Weather&time=7Day

share|improve this answer

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

share|improve this answer

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges

answered Mar 26 at 1:04

hoangdvhoangdv

2,83011 gold badge77 silver badges1919 bronze badges