Convert Json array into csv using apache NifiCreate ArrayList from arrayHow do I check if an array includes an object in JavaScript?Can comments be used in JSON?How to append something to an array?How can I pretty-print JSON in a shell script?What is the correct JSON content type?Why does Google prepend while(1); to their JSON responses?Loop through an array in JavaScriptHow do I remove a particular element from an array in JavaScript?For-each over an array in JavaScript?
Has the United States ever had a non-Christian President?
How can a hefty sand storm happen in a thin atmosphere like Martian?
Why does blending blueberries, milk, banana and vanilla extract cause the mixture to have a yogurty consistency?
Blender 2.80 Remove double vertices option gone
Can full drive backup be used instead of MSSQL database backup?
Can I hide the part of long lines that exceeds the visual line?
Why are oscilloscope input impedances so low?
My first C++ game (snake console game)
Why is "breaking the mould" positively connoted?
Is there a word that describes the unjustified use of a more complex word?
Would a "Permanence" spell in 5e be overpowered?
Undefined Control Sequence for my 0-norm
Why would a military not separate its forces into different branches?
Would a small hole in a Faraday cage drastically reduce its effectiveness at blocking interference?
Why does sound not move through a wall?
Is it normal for gliders not to have attitude indicators?
Is any special diet an effective treatment of autism?
Will a God Eternal enchanted with Deep Freeze shuffle back into the deck if it dies?
How to pass query parameters in URL in Salesforce Summer 19 Release?
In Futurama, how many beings has Leela slept with?
Where to draw the line between quantum mechanics theory and its interpretation(s)?
Endgame puzzle: How to avoid stalemate and win?
Which US defense organization would respond to an invasion like this?
GitLab account hacked and repo wiped
Convert Json array into csv using apache Nifi
Create ArrayList from arrayHow do I check if an array includes an object in JavaScript?Can comments be used in JSON?How to append something to an array?How can I pretty-print JSON in a shell script?What is the correct JSON content type?Why does Google prepend while(1); to their JSON responses?Loop through an array in JavaScriptHow do I remove a particular element from an array in JavaScript?For-each over an array in JavaScript?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I'm looking to convert JSON with an array to csv format. The number of elements inside the array is dynamic for each row. I tried using this flow, ( attached the flow file xml on the post ).
GetFile --> ConvertRecord --> UpdateAttribute --> PutFile
Are there any other alternatives?
JSON format:
"LogData":
"Location": "APAC",
"product": "w1" , "Outcome": [
"limit": "0",
"pri": "3",
"result": "pass"
,
"limit": "1",
"pri": "2",
"result": "pass"
,
"limit": "5",
"priority": "1",
"result": "fail"
], "attr":
"vers": "1",
"datetime": "2018-01-10 00:36:00"
Expected output in csv:
location, product, limit, pri, result, vers, datetime
APAC w1 0 3 pass 1 2018-01-10 00:36:00
APAC w1 1 2 pass 1 2018-01-10 00:36:00
APAC w1 5 1 fail 1 2018-01-10 00:36:00
Output from the attached flow:
LogData,Outcome,attr
"MapRecord[product=w1, Location=APAC]","[MapRecord[limit=0, result=pass, pri=3], MapRecord[limit=1, result=pass, pri=2], MapRecord[limit=5, result=fail]]","MapRecord[datetime=2018-01-10 00:36:00, vers=1]"
ConvertRecord -- I am using JSONTreereader and CSVRecordSSetwriter configurations as below:
JSONTreereader Controler service config:
CSVRecordSetwriter controller service config:
AvroschemaRegistry Controller service config:
Avro schema :
"name": "myschema", "type": "record", "namespace": "myschema", "fields": ["name": "LogData","type": "name": "LogData", "type": "record", "fields": [ "name": "Location", "type": "string", "name": "product", "type": "string" ],"name": "Outcome","type": "type": "array", "items": "name": "Outcome_record","type": "record","fields": [ "name": "limit","type": "string" , "name": "pri","type": ["string","null"] , "name": "result","type": "string" ] ,"name": "attr","type": "name": "attr", "type": "record", "fields": [ "name": "vers", "type": "string", "name": "datetime", "type": "string" ] ]
arrays json apache-nifi jsonpath
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
add a comment |
I'm looking to convert JSON with an array to csv format. The number of elements inside the array is dynamic for each row. I tried using this flow, ( attached the flow file xml on the post ).
GetFile --> ConvertRecord --> UpdateAttribute --> PutFile
Are there any other alternatives?
JSON format:
"LogData":
"Location": "APAC",
"product": "w1" , "Outcome": [
"limit": "0",
"pri": "3",
"result": "pass"
,
"limit": "1",
"pri": "2",
"result": "pass"
,
"limit": "5",
"priority": "1",
"result": "fail"
], "attr":
"vers": "1",
"datetime": "2018-01-10 00:36:00"
Expected output in csv:
location, product, limit, pri, result, vers, datetime
APAC w1 0 3 pass 1 2018-01-10 00:36:00
APAC w1 1 2 pass 1 2018-01-10 00:36:00
APAC w1 5 1 fail 1 2018-01-10 00:36:00
Output from the attached flow:
LogData,Outcome,attr
"MapRecord[product=w1, Location=APAC]","[MapRecord[limit=0, result=pass, pri=3], MapRecord[limit=1, result=pass, pri=2], MapRecord[limit=5, result=fail]]","MapRecord[datetime=2018-01-10 00:36:00, vers=1]"
ConvertRecord -- I am using JSONTreereader and CSVRecordSSetwriter configurations as below:
JSONTreereader Controler service config:
CSVRecordSetwriter controller service config:
AvroschemaRegistry Controller service config:
Avro schema :
"name": "myschema", "type": "record", "namespace": "myschema", "fields": ["name": "LogData","type": "name": "LogData", "type": "record", "fields": [ "name": "Location", "type": "string", "name": "product", "type": "string" ],"name": "Outcome","type": "type": "array", "items": "name": "Outcome_record","type": "record","fields": [ "name": "limit","type": "string" , "name": "pri","type": ["string","null"] , "name": "result","type": "string" ] ,"name": "attr","type": "name": "attr", "type": "record", "fields": [ "name": "vers", "type": "string", "name": "datetime", "type": "string" ] ]
arrays json apache-nifi jsonpath
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
add a comment |
I'm looking to convert JSON with an array to csv format. The number of elements inside the array is dynamic for each row. I tried using this flow, ( attached the flow file xml on the post ).
GetFile --> ConvertRecord --> UpdateAttribute --> PutFile
Are there any other alternatives?
JSON format:
"LogData":
"Location": "APAC",
"product": "w1" , "Outcome": [
"limit": "0",
"pri": "3",
"result": "pass"
,
"limit": "1",
"pri": "2",
"result": "pass"
,
"limit": "5",
"priority": "1",
"result": "fail"
], "attr":
"vers": "1",
"datetime": "2018-01-10 00:36:00"
Expected output in csv:
location, product, limit, pri, result, vers, datetime
APAC w1 0 3 pass 1 2018-01-10 00:36:00
APAC w1 1 2 pass 1 2018-01-10 00:36:00
APAC w1 5 1 fail 1 2018-01-10 00:36:00
Output from the attached flow:
LogData,Outcome,attr
"MapRecord[product=w1, Location=APAC]","[MapRecord[limit=0, result=pass, pri=3], MapRecord[limit=1, result=pass, pri=2], MapRecord[limit=5, result=fail]]","MapRecord[datetime=2018-01-10 00:36:00, vers=1]"
ConvertRecord -- I am using JSONTreereader and CSVRecordSSetwriter configurations as below:
JSONTreereader Controler service config:
CSVRecordSetwriter controller service config:
AvroschemaRegistry Controller service config:
Avro schema :
"name": "myschema", "type": "record", "namespace": "myschema", "fields": ["name": "LogData","type": "name": "LogData", "type": "record", "fields": [ "name": "Location", "type": "string", "name": "product", "type": "string" ],"name": "Outcome","type": "type": "array", "items": "name": "Outcome_record","type": "record","fields": [ "name": "limit","type": "string" , "name": "pri","type": ["string","null"] , "name": "result","type": "string" ] ,"name": "attr","type": "name": "attr", "type": "record", "fields": [ "name": "vers", "type": "string", "name": "datetime", "type": "string" ] ]
arrays json apache-nifi jsonpath
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
I'm looking to convert JSON with an array to csv format. The number of elements inside the array is dynamic for each row. I tried using this flow, ( attached the flow file xml on the post ).
GetFile --> ConvertRecord --> UpdateAttribute --> PutFile
Are there any other alternatives?
JSON format:
"LogData":
"Location": "APAC",
"product": "w1" , "Outcome": [
"limit": "0",
"pri": "3",
"result": "pass"
,
"limit": "1",
"pri": "2",
"result": "pass"
,
"limit": "5",
"priority": "1",
"result": "fail"
], "attr":
"vers": "1",
"datetime": "2018-01-10 00:36:00"
Expected output in csv:
location, product, limit, pri, result, vers, datetime
APAC w1 0 3 pass 1 2018-01-10 00:36:00
APAC w1 1 2 pass 1 2018-01-10 00:36:00
APAC w1 5 1 fail 1 2018-01-10 00:36:00
Output from the attached flow:
LogData,Outcome,attr
"MapRecord[product=w1, Location=APAC]","[MapRecord[limit=0, result=pass, pri=3], MapRecord[limit=1, result=pass, pri=2], MapRecord[limit=5, result=fail]]","MapRecord[datetime=2018-01-10 00:36:00, vers=1]"
ConvertRecord -- I am using JSONTreereader and CSVRecordSSetwriter configurations as below:
JSONTreereader Controler service config:
CSVRecordSetwriter controller service config:
AvroschemaRegistry Controller service config:
Avro schema :
"name": "myschema", "type": "record", "namespace": "myschema", "fields": ["name": "LogData","type": "name": "LogData", "type": "record", "fields": [ "name": "Location", "type": "string", "name": "product", "type": "string" ],"name": "Outcome","type": "type": "array", "items": "name": "Outcome_record","type": "record","fields": [ "name": "limit","type": "string" , "name": "pri","type": ["string","null"] , "name": "result","type": "string" ] ,"name": "attr","type": "name": "attr", "type": "record", "fields": [ "name": "vers", "type": "string", "name": "datetime", "type": "string" ] ]
arrays json apache-nifi jsonpath
arrays json apache-nifi jsonpath
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
asked Mar 23 at 2:57
vishwanatha handadivishwanatha handadi
124
124
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Try this spec in JoltTransformJSON before ConvertRecord:
"operation": "shift",
"spec":
"Outcome":
"*":
"@(3,LogData.Location)": "[#2].location",
"@(3,LogData.product)": "[#2].product",
"@(3,attr.vers)": "[#2].vers",
"@(3,attr.datetime)": "[#2].datetime",
"*": "[#2].&"
]```
answered Mar 25 at 23:27
mattybmattyb
8,1411018
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
add a comment |
Seems that you need to performa JoltTransform before convert to CSV, if not is not going to work.
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55310186%2fconvert-json-array-into-csv-using-apache-nifi%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Try this spec in JoltTransformJSON before ConvertRecord:
"operation": "shift",
"spec":
"Outcome":
"*":
"@(3,LogData.Location)": "[#2].location",
"@(3,LogData.product)": "[#2].product",
"@(3,attr.vers)": "[#2].vers",
"@(3,attr.datetime)": "[#2].datetime",
"*": "[#2].&"
]```
answered Mar 25 at 23:27
mattybmattyb
8,1411018
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
add a comment |
Try this spec in JoltTransformJSON before ConvertRecord:
"operation": "shift",
"spec":
"Outcome":
"*":
"@(3,LogData.Location)": "[#2].location",
"@(3,LogData.product)": "[#2].product",
"@(3,attr.vers)": "[#2].vers",
"@(3,attr.datetime)": "[#2].datetime",
"*": "[#2].&"
]```
answered Mar 25 at 23:27
mattybmattyb
8,1411018
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
add a comment |
Try this spec in JoltTransformJSON before ConvertRecord:
"operation": "shift",
"spec":
"Outcome":
"*":
"@(3,LogData.Location)": "[#2].location",
"@(3,LogData.product)": "[#2].product",
"@(3,attr.vers)": "[#2].vers",
"@(3,attr.datetime)": "[#2].datetime",
"*": "[#2].&"
]```
answered Mar 25 at 23:27
mattybmattyb
8,1411018
Try this spec in JoltTransformJSON before ConvertRecord:
"operation": "shift",
"spec":
"Outcome":
"*":
"@(3,LogData.Location)": "[#2].location",
"@(3,LogData.product)": "[#2].product",
"@(3,attr.vers)": "[#2].vers",
"@(3,attr.datetime)": "[#2].datetime",
"*": "[#2].&"
]```
answered Mar 25 at 23:27
mattybmattyb
8,1411018
answered Mar 25 at 23:27
mattybmattyb
8,1411018
answered Mar 25 at 23:27
mattybmattyb
8,1411018
answered Mar 25 at 23:27
mattybmattyb
8,1411018
8,1411018
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
add a comment |
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Thanks a ton @mattyb for the help. This works like a charm. One question: If i have 1300 + array elements inside Outcome:[], will the jolt transform works well or will there be any overhead while converting 1 incoming json record to 1300+ json records?
– vishwanatha handadi
Mar 27 at 13:58
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
Not sure what you mean by overhead, but I can tell you that all 1300+ flow files will be transferred at the same time, once they are all done processing and the session is committed.
– mattyb
Mar 29 at 19:28
add a comment |
Seems that you need to performa JoltTransform before convert to CSV, if not is not going to work.
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
add a comment |
Seems that you need to performa JoltTransform before convert to CSV, if not is not going to work.
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
add a comment |
Seems that you need to performa JoltTransform before convert to CSV, if not is not going to work.
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
Seems that you need to performa JoltTransform before convert to CSV, if not is not going to work.
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
answered Mar 24 at 12:17
Óscar AndreuÓscar Andreu
1,176728
1,176728
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55310186%2fconvert-json-array-into-csv-using-apache-nifi%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
