Scraping Data using R - Form with List in POSTConvert form data to JavaScript object with jQueryWhy can't Python parse this JSON data?How to process POST data in Node.js?How do I manually fire HTTP POST requests with Firefox or Chrome?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?x-www-form-urlencoded Vs json HTTP POSTC# HttpWebRequest POST => !! Unexpected error while processing request: invalid %-encodingHow to pass json POST data to Web API method as an object?How to send JSON data in body with header using post method in PHP(cURL)“Missing grant type” Oauth from R using httr package
Using "subway" as name for London Underground?
How do governments keep track of their issued currency?
Arriving at the same result with the opposite hypotheses
Determining fair price for profitable mobile app business
Can Rydberg constant be in joules?
Pathfinder warbow concept review
Union with anonymous struct with flexible array member
How is John Wick 3 a 15 certificate?
Cascading Switches. Will it affect performance?
Generate basis elements of the Steenrod algebra
How to manually rewind film?
How to trick the reader into thinking they're following a redshirt instead of the protagonist?
How come the nude protesters were not arrested?
You have (3^2 + 2^3 + 2^2) Guesses Left. Figure out the Last one
What ways have you found to get edits from non-LaTeX users?
Medieval flying castle propulsion
How to handle self harm scars on the arm in work environment?
Alternate way of computing the probability of being dealt a 13 card hand with 3 kings given that you have been dealt 2 kings
How did old MS-DOS games utilize various graphic cards?
English word for "product of tinkering"
How is water heavier than petrol, even though its molecular weight is less than petrol?
Thread Pool C++ Implementation
Were Alexander the Great and Hephaestion lovers?
How to use memset in c++?
Scraping Data using R - Form with List in POST
Convert form data to JavaScript object with jQueryWhy can't Python parse this JSON data?How to process POST data in Node.js?How do I manually fire HTTP POST requests with Firefox or Chrome?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?x-www-form-urlencoded Vs json HTTP POSTC# HttpWebRequest POST => !! Unexpected error while processing request: invalid %-encodingHow to pass json POST data to Web API method as an object?How to send JSON data in body with header using post method in PHP(cURL)“Missing grant type” Oauth from R using httr package
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
"section":[1,4],"period":20170501
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
r json web-scraping http-post httr
asked Mar 24 at 18:06
ChrisChris
53311332
add a comment |
I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
"section":[1,4],"period":20170501
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
r json web-scraping http-post httr
asked Mar 24 at 18:06
ChrisChris
53311332
add a comment |
I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
"section":[1,4],"period":20170501
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
r json web-scraping http-post httr
asked Mar 24 at 18:06
ChrisChris
53311332
I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
"section":[1,4],"period":20170501
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
r json web-scraping http-post httr
r json web-scraping http-post httr
asked Mar 24 at 18:06
ChrisChris
53311332
asked Mar 24 at 18:06
ChrisChris
53311332
edited Mar 27 at 20:21
Chris
asked Mar 24 at 18:06
ChrisChris
53311332
asked Mar 24 at 18:06
ChrisChris
53311332
asked Mar 24 at 18:06
ChrisChris
53311332
53311332
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
answered Mar 29 at 19:56
ChrisChris
53311332
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55326872%2fscraping-data-using-r-form-with-list-in-post%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
answered Mar 29 at 19:56
ChrisChris
53311332
add a comment |
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
answered Mar 29 at 19:56
ChrisChris
53311332
add a comment |
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
answered Mar 29 at 19:56
ChrisChris
53311332
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
answered Mar 29 at 19:56
ChrisChris
53311332
edited Mar 29 at 21:24
answered Mar 29 at 19:56
ChrisChris
53311332
answered Mar 29 at 19:56
ChrisChris
53311332
answered Mar 29 at 19:56
ChrisChris
53311332
53311332
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55326872%2fscraping-data-using-r-form-with-list-in-post%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
