Flattened JSON Repeating Columns Pandas Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Selecting multiple columns in a pandas dataframeRenaming columns in pandasAdding new column to existing DataFrame in Python pandasDelete column from pandas DataFrame by column name“Large data” work flows using pandasChange data type of columns in PandasHow to iterate over rows in a DataFrame in Pandas?Select rows from a DataFrame based on values in a column in pandasConvert Python dict into a dataframeGet list from pandas DataFrame column headers
Putting Ant-Man on house arrest
Etymology of 見舞い
Can I take recommendation from someone I met at a conference?
Can Deduction Guide have an explicit(bool) specifier?
"Destructive force" carried by a B-52?
Weaponising the Grasp-at-a-Distance spell
How to mute a string and play another at the same time
Can a Knight grant Knighthood to another?
How do I deal with an erroneously large refund?
Is "ein Herz wie das meine" an antiquated or colloquial use of the possesive pronoun?
“Since the train was delayed for more than an hour, passengers were given a full refund.” – Why is there no article before “passengers”?
Why did Europeans not widely domesticate foxes?
Coin Game with infinite paradox
xkeyval -- read keys from file
Short story about an alien named Ushtu(?) coming from a future Earth, when ours was destroyed by a nuclear explosion
Why does BitLocker not use RSA?
Why doesn't the university give past final exams' answers?
Lights are flickering on and off after accidentally bumping into light switch
Can the van der Waals coefficients be negative in the van der Waals equation for real gases?
Lemmatization Vs Stemming
Raising a bilingual kid. When should we introduce the majority language?
What's the connection between Mr. Nancy and fried chicken?
Compiling and throwing simple dynamic exceptions at runtime for JVM
Why isn't everyone flabbergasted about Bran's "gift"?
Flattened JSON Repeating Columns Pandas
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!Selecting multiple columns in a pandas dataframeRenaming columns in pandasAdding new column to existing DataFrame in Python pandasDelete column from pandas DataFrame by column name“Large data” work flows using pandasChange data type of columns in PandasHow to iterate over rows in a DataFrame in Pandas?Select rows from a DataFrame based on values in a column in pandasConvert Python dict into a dataframeGet list from pandas DataFrame column headers
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I'm having an issue currently with pulling from an API, getting a JSON dict, then flattening it, and placing it into a dataframe.
The data is structured like this from the json:
X1_0, X2_0, X3_0 ... X1_1, X2_1, X2_1, ... X1_2, X2_2, X2_3
and when I flatten it and place into a dataframe I get each flattened key as an individual column header rather than all combined since they have they _#.
So rather than getting something that's shape is 22 x 6 I get something that would be like 1 x 130.
I'm basically just interested in getting the shape of the dataframe correct but I'm not sure how I should fix it, and whether it should be done before flattening or after?
Any help is appreciated
python pandas
asked Mar 22 at 14:00
nillamnillam
113
add a comment |
I'm having an issue currently with pulling from an API, getting a JSON dict, then flattening it, and placing it into a dataframe.
The data is structured like this from the json:
X1_0, X2_0, X3_0 ... X1_1, X2_1, X2_1, ... X1_2, X2_2, X2_3
and when I flatten it and place into a dataframe I get each flattened key as an individual column header rather than all combined since they have they _#.
So rather than getting something that's shape is 22 x 6 I get something that would be like 1 x 130.
I'm basically just interested in getting the shape of the dataframe correct but I'm not sure how I should fix it, and whether it should be done before flattening or after?
Any help is appreciated
python pandas
asked Mar 22 at 14:00
nillamnillam
113
Why don't you change your flattening code to produce a JSON array with six items[X1, X2, X3, X4, X5...]?
– Aaron Digulla
Mar 22 at 14:36
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40
add a comment |
I'm having an issue currently with pulling from an API, getting a JSON dict, then flattening it, and placing it into a dataframe.
The data is structured like this from the json:
X1_0, X2_0, X3_0 ... X1_1, X2_1, X2_1, ... X1_2, X2_2, X2_3
and when I flatten it and place into a dataframe I get each flattened key as an individual column header rather than all combined since they have they _#.
So rather than getting something that's shape is 22 x 6 I get something that would be like 1 x 130.
I'm basically just interested in getting the shape of the dataframe correct but I'm not sure how I should fix it, and whether it should be done before flattening or after?
Any help is appreciated
python pandas
asked Mar 22 at 14:00
nillamnillam
113
I'm having an issue currently with pulling from an API, getting a JSON dict, then flattening it, and placing it into a dataframe.
The data is structured like this from the json:
X1_0, X2_0, X3_0 ... X1_1, X2_1, X2_1, ... X1_2, X2_2, X2_3
and when I flatten it and place into a dataframe I get each flattened key as an individual column header rather than all combined since they have they _#.
So rather than getting something that's shape is 22 x 6 I get something that would be like 1 x 130.
I'm basically just interested in getting the shape of the dataframe correct but I'm not sure how I should fix it, and whether it should be done before flattening or after?
Any help is appreciated
python pandas
python pandas
asked Mar 22 at 14:00
nillamnillam
113
asked Mar 22 at 14:00
nillamnillam
113
edited Mar 22 at 14:12
nillam
asked Mar 22 at 14:00
nillamnillam
113
asked Mar 22 at 14:00
nillamnillam
113
asked Mar 22 at 14:00
nillamnillam
113
113
Why don't you change your flattening code to produce a JSON array with six items[X1, X2, X3, X4, X5...]?
– Aaron Digulla
Mar 22 at 14:36
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40
add a comment |
Why don't you change your flattening code to produce a JSON array with six items[X1, X2, X3, X4, X5...]?
– Aaron Digulla
Mar 22 at 14:36
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40
Why don't you change your flattening code to produce a JSON array with six items
[X1, X2, X3, X4, X5...]?– Aaron Digulla
Mar 22 at 14:36
Why don't you change your flattening code to produce a JSON array with six items
[X1, X2, X3, X4, X5...]?– Aaron Digulla
Mar 22 at 14:36
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40
add a comment |
1 Answer
1
active
oldest
votes
Try to strip the _# from the keys. My guess is that they are added in the flattening step, so it should be easy to get rid of them.
Now you will get several values per key. Fix this by creating a list of JSON objects where each contains all the values for each key with the same _#.
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55301270%2fflattened-json-repeating-columns-pandas%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Try to strip the _# from the keys. My guess is that they are added in the flattening step, so it should be easy to get rid of them.
Now you will get several values per key. Fix this by creating a list of JSON objects where each contains all the values for each key with the same _#.
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
add a comment |
Try to strip the _# from the keys. My guess is that they are added in the flattening step, so it should be easy to get rid of them.
Now you will get several values per key. Fix this by creating a list of JSON objects where each contains all the values for each key with the same _#.
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
add a comment |
Try to strip the _# from the keys. My guess is that they are added in the flattening step, so it should be easy to get rid of them.
Now you will get several values per key. Fix this by creating a list of JSON objects where each contains all the values for each key with the same _#.
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
Try to strip the _# from the keys. My guess is that they are added in the flattening step, so it should be easy to get rid of them.
Now you will get several values per key. Fix this by creating a list of JSON objects where each contains all the values for each key with the same _#.
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
answered Mar 22 at 14:52
Aaron DigullaAaron Digulla
251k87482701
251k87482701
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55301270%2fflattened-json-repeating-columns-pandas%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Why don't you change your flattening code to produce a JSON array with six items
[X1, X2, X3, X4, X5...]?– Aaron Digulla
Mar 22 at 14:36
I just used the 6 as an example but in reality when the input changes (in this case lat/long for Bing Maps API), the output will change and could be more/less than 6 items. So a static fix like setting the array to six items wouldn't really work because sometimes the api may return 4, 8, or even 100. I'm thinking now the best option might be to remove the identifying number from the columns since that seems to be the only reason they're not stacking
– nillam
Mar 22 at 14:40