How to flatten json properties to _source used ELK?How does database indexing work?How do I format a Microsoft JSON date?Can comments be used in JSON?How can I pretty-print JSON in a shell script?What is the correct JSON content type?How to make a flat list out of list of listsWhy does Google prepend while(1); to their JSON responses?How can I pretty-print JSON using JavaScript?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?Merge/flatten an array of arrays
How can I discourage sharing internal API keys within a company?
Why is the T-1000 humanoid?
Why is the Digital 0 not 0V in computer systems?
Can I toggle Do Not Disturb on/off on my Mac as easily as I can on my iPhone?
How to stabilise the bicycle seatpost and saddle when it is all the way up?
Diffraction of a wave passing through double slits
What does a Light weapon mean mechanically?
Sort files in a given folders and provide as a list
What is and what isn't ullage in rocket science?
How do I say "quirky" in German without sounding derogatory?
How are aircraft depainted?
What exactly is a marshrutka (маршрутка)?
My research paper filed as a patent in China by my Chinese supervisor without me as inventor
How do email clients "send later" without storing a password?
Can I conceal an antihero's insanity - and should I?
Parallel resistance in electric circuits
How would you control supersoldiers in a late iron-age society?
Confirm the ending of a string
POSIX compatible way to get user name associated with a user ID
What is the derivative of an exponential function with another function as its base?
Were Roman public roads build by private companies?
Why is this weapon searching for a new owner?
How are unbalanced coaxial cables used for broadcasting TV signals without any problems?
Why is template constructor preferred to copy constructor?
How to flatten json properties to _source used ELK?
How does database indexing work?How do I format a Microsoft JSON date?Can comments be used in JSON?How can I pretty-print JSON in a shell script?What is the correct JSON content type?How to make a flat list out of list of listsWhy does Google prepend while(1); to their JSON responses?How can I pretty-print JSON using JavaScript?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?Merge/flatten an array of arrays
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
Suppose I have these 2 log messages send to logstash/elasticsearch:
"events": [
"Message": "Get all motors",
"Level": "Information"
,
"Message": "Get all motors",
"Level": "Information"
]
The expected search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
]
I used below logstash config:
# Http input listening port 8080
input
http
#default host 0.0.0.0:8080
codec => json
# Separate the logs
filter
split
field => "events"
target => "e"
mutate
remove_field => ["events", "headers"]
# Send the logs to Elasticsearch
output
elasticsearch
hosts => "localhost:9200"
index=>"mylog-%+YYYY.MM.dd"
document_type => "log"
The actual search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
]
The different is:
- In the actual result, my log info is nested in
eproperty - I want all the log info props flattened (direct children of
_source)
Elasticsearch document suggest remove target from filter:
filter
split
field => "events"
But if I do that, my result will be empty:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
]
How can I achieve the expected result?
My environment:
- Windows 10 x64 Pro
- ELK 6.7
- JDK 8u201
json indexing elastic-stack flatten
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
add a comment
|
Suppose I have these 2 log messages send to logstash/elasticsearch:
"events": [
"Message": "Get all motors",
"Level": "Information"
,
"Message": "Get all motors",
"Level": "Information"
]
The expected search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
]
I used below logstash config:
# Http input listening port 8080
input
http
#default host 0.0.0.0:8080
codec => json
# Separate the logs
filter
split
field => "events"
target => "e"
mutate
remove_field => ["events", "headers"]
# Send the logs to Elasticsearch
output
elasticsearch
hosts => "localhost:9200"
index=>"mylog-%+YYYY.MM.dd"
document_type => "log"
The actual search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
]
The different is:
- In the actual result, my log info is nested in
eproperty - I want all the log info props flattened (direct children of
_source)
Elasticsearch document suggest remove target from filter:
filter
split
field => "events"
But if I do that, my result will be empty:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
]
How can I achieve the expected result?
My environment:
- Windows 10 x64 Pro
- ELK 6.7
- JDK 8u201
json indexing elastic-stack flatten
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
add a comment
|
Suppose I have these 2 log messages send to logstash/elasticsearch:
"events": [
"Message": "Get all motors",
"Level": "Information"
,
"Message": "Get all motors",
"Level": "Information"
]
The expected search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
]
I used below logstash config:
# Http input listening port 8080
input
http
#default host 0.0.0.0:8080
codec => json
# Separate the logs
filter
split
field => "events"
target => "e"
mutate
remove_field => ["events", "headers"]
# Send the logs to Elasticsearch
output
elasticsearch
hosts => "localhost:9200"
index=>"mylog-%+YYYY.MM.dd"
document_type => "log"
The actual search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
]
The different is:
- In the actual result, my log info is nested in
eproperty - I want all the log info props flattened (direct children of
_source)
Elasticsearch document suggest remove target from filter:
filter
split
field => "events"
But if I do that, my result will be empty:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
]
How can I achieve the expected result?
My environment:
- Windows 10 x64 Pro
- ELK 6.7
- JDK 8u201
json indexing elastic-stack flatten
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
Suppose I have these 2 log messages send to logstash/elasticsearch:
"events": [
"Message": "Get all motors",
"Level": "Information"
,
"Message": "Get all motors",
"Level": "Information"
]
The expected search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"Message" : "Get all motors",
"Level" : "Information"
]
I used below logstash config:
# Http input listening port 8080
input
http
#default host 0.0.0.0:8080
codec => json
# Separate the logs
filter
split
field => "events"
target => "e"
mutate
remove_field => ["events", "headers"]
# Send the logs to Elasticsearch
output
elasticsearch
hosts => "localhost:9200"
index=>"mylog-%+YYYY.MM.dd"
document_type => "log"
The actual search result:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1",
"e" :
"Message" : "Get all motors",
"Level" : "Information"
]
The different is:
- In the actual result, my log info is nested in
eproperty - I want all the log info props flattened (direct children of
_source)
Elasticsearch document suggest remove target from filter:
filter
split
field => "events"
But if I do that, my result will be empty:
"hits" :
"total" : 2,
"max_score" : 1.0,
"hits" : [
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
,
"_source" :
"@timestamp" : "2019-03-28T10:05:05.649Z",
"@version" : "1"
]
How can I achieve the expected result?
My environment:
- Windows 10 x64 Pro
- ELK 6.7
- JDK 8u201
json indexing elastic-stack flatten
json indexing elastic-stack flatten
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
edited Mar 28 at 11:59
Tuyen Pham
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
asked Mar 28 at 10:29
Tuyen PhamTuyen Pham
4,87610 gold badges44 silver badges85 bronze badges
4,87610 gold badges44 silver badges85 bronze badges
add a comment
|
add a comment
|
1 Answer
1
active
oldest
votes
remove the target => "e" part, your target is the root of the document.
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
add a comment
|
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55395337%2fhow-to-flatten-json-properties-to-source-used-elk%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
remove the target => "e" part, your target is the root of the document.
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
add a comment
|
remove the target => "e" part, your target is the root of the document.
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
add a comment
|
remove the target => "e" part, your target is the root of the document.
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
remove the target => "e" part, your target is the root of the document.
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
answered Mar 28 at 10:31
Vincent ChalmelVincent Chalmel
3342 silver badges14 bronze badges
3342 silver badges14 bronze badges
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
add a comment
|
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
I tried that. If I remove it, the log info will not be recorded at all
– Tuyen Pham
Mar 28 at 10:39
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
Sorry, I was not clear enough, don't remove the line, but make it target the root of the document, not "e"
– Vincent Chalmel
Mar 28 at 13:24
add a comment
|
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55395337%2fhow-to-flatten-json-properties-to-source-used-elk%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
