Forward onpremise logs to Kinesis FirehoseCode in Node.js AWS Lambda Package will not call putRecord() for Adding Data into AWS Kinesis Firehose StreamKinesis input stream into LogstashFirehose to S3: What happens to data after unsuccessful tries for 24 hoursIssues with streaming data to AWS Kinesis Firehose from PythonKinesis Analytics Destination Guidance: Lambda vs Kinesis Stream to LambdaKinesis Agent reports error from python stream dataSetting up write access of s3 owned by one aws account to firehose stream owned by another aws accountKinesis FireHose vs Kinesis Data Stream loosing dataResourceNotFoundException while adding data to Kinesis Firehose stream using Lambda
Does this Wild Magic result affect the sorcerer or just other creatures?
Would it be a copyright violation if I made a character’s full name refer to a song?
Old sci-fi story: radiation mutated the animals, a boy loses a limb, but it's okay because "humans used to do great with only two arms"
How does a pilot select the correct ILS when the airport has parallel runways?
Why do all the teams that I have worked with always finish a sprint without completion of all the stories?
Did the CIA blow up a Siberian pipeline in 1982?
Why use cross notes in sheet music for hip hop tracks?
Why do some professors with PhDs leave their professorships to teach high school?
Output of "$OSTYPE:6" on old releases of Mac OS X
What size of powerbank will I need to power a phone and DSLR for 2 weeks?
When to remove insignificant variables?
Really Old Stock Valuation
Can White Castle?
Why does the Saturn V have standalone inter-stage rings?
Unusual mail headers, evidence of an attempted attack. Have I been pwned?
How do I set an alias to a terminal line?
Dates on degrees don’t make sense – will people care?
Is there a maximum distance from a planet that a moon can orbit?
What did River say when she woke from her proto-comatose state?
Why do even high-end cameras often still include normal (non-cross-type) AF sensors?
Trainee keeps missing deadlines for independent learning
JSON selector class in Python
What can I do with a research project that is my university’s intellectual property?
What reason would an alien civilization have for building a Dyson Sphere (or Swarm) if cheap Nuclear fusion is available?
Forward onpremise logs to Kinesis Firehose
Code in Node.js AWS Lambda Package will not call putRecord() for Adding Data into AWS Kinesis Firehose StreamKinesis input stream into LogstashFirehose to S3: What happens to data after unsuccessful tries for 24 hoursIssues with streaming data to AWS Kinesis Firehose from PythonKinesis Analytics Destination Guidance: Lambda vs Kinesis Stream to LambdaKinesis Agent reports error from python stream dataSetting up write access of s3 owned by one aws account to firehose stream owned by another aws accountKinesis FireHose vs Kinesis Data Stream loosing dataResourceNotFoundException while adding data to Kinesis Firehose stream using Lambda
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
For my application logging I am planning to forward the log from my on-premise servers to AWS Kinesis firehose. I am trying to use Logstash and the log-stash-output-plugin
https://github.com/samcday/logstash-output-kinesis
But I think this plugin requires to forward the log data to Kinesis Data Stream and then we can create a pipe to Kinesis Firehose. If we point to the Kinesis Firehose Delivery stream then we are getting the error -
ResourceNotFoundException Message: Stream test under account XXXX not found
But if we point the log-stash to Kinesis data stream and then pipe it to Kinesis Firehose then it is working fine.
Have anyone used this log-stashoutput plugin to forward the log data directly to Kinesis Firehose.
amazon-kinesis amazon-kinesis-firehose
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
add a comment |
For my application logging I am planning to forward the log from my on-premise servers to AWS Kinesis firehose. I am trying to use Logstash and the log-stash-output-plugin
https://github.com/samcday/logstash-output-kinesis
But I think this plugin requires to forward the log data to Kinesis Data Stream and then we can create a pipe to Kinesis Firehose. If we point to the Kinesis Firehose Delivery stream then we are getting the error -
ResourceNotFoundException Message: Stream test under account XXXX not found
But if we point the log-stash to Kinesis data stream and then pipe it to Kinesis Firehose then it is working fine.
Have anyone used this log-stashoutput plugin to forward the log data directly to Kinesis Firehose.
amazon-kinesis amazon-kinesis-firehose
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
add a comment |
For my application logging I am planning to forward the log from my on-premise servers to AWS Kinesis firehose. I am trying to use Logstash and the log-stash-output-plugin
https://github.com/samcday/logstash-output-kinesis
But I think this plugin requires to forward the log data to Kinesis Data Stream and then we can create a pipe to Kinesis Firehose. If we point to the Kinesis Firehose Delivery stream then we are getting the error -
ResourceNotFoundException Message: Stream test under account XXXX not found
But if we point the log-stash to Kinesis data stream and then pipe it to Kinesis Firehose then it is working fine.
Have anyone used this log-stashoutput plugin to forward the log data directly to Kinesis Firehose.
amazon-kinesis amazon-kinesis-firehose
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
For my application logging I am planning to forward the log from my on-premise servers to AWS Kinesis firehose. I am trying to use Logstash and the log-stash-output-plugin
https://github.com/samcday/logstash-output-kinesis
But I think this plugin requires to forward the log data to Kinesis Data Stream and then we can create a pipe to Kinesis Firehose. If we point to the Kinesis Firehose Delivery stream then we are getting the error -
ResourceNotFoundException Message: Stream test under account XXXX not found
But if we point the log-stash to Kinesis data stream and then pipe it to Kinesis Firehose then it is working fine.
Have anyone used this log-stashoutput plugin to forward the log data directly to Kinesis Firehose.
amazon-kinesis amazon-kinesis-firehose
amazon-kinesis amazon-kinesis-firehose
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
edited Mar 25 at 10:53
bugfreerammohan
1,3451215
1,3451215
asked Mar 25 at 8:43
DeveloperDeveloper
388
asked Mar 25 at 8:43
DeveloperDeveloper
388
asked Mar 25 at 8:43
DeveloperDeveloper
388
388
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly.
- https://github.com/awslabs/amazon-kinesis-producer/issues/29
Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint.
So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException".
- https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55
answered Mar 27 at 23:55
HarshdevHarshdev
361
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55333985%2fforward-onpremise-logs-to-kinesis-firehose%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly.
- https://github.com/awslabs/amazon-kinesis-producer/issues/29
Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint.
So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException".
- https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55
answered Mar 27 at 23:55
HarshdevHarshdev
361
add a comment |
logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly.
- https://github.com/awslabs/amazon-kinesis-producer/issues/29
Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint.
So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException".
- https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55
answered Mar 27 at 23:55
HarshdevHarshdev
361
add a comment |
logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly.
- https://github.com/awslabs/amazon-kinesis-producer/issues/29
Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint.
So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException".
- https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55
answered Mar 27 at 23:55
HarshdevHarshdev
361
logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly.
- https://github.com/awslabs/amazon-kinesis-producer/issues/29
Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint.
So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException".
- https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55
answered Mar 27 at 23:55
HarshdevHarshdev
361
answered Mar 27 at 23:55
HarshdevHarshdev
361
answered Mar 27 at 23:55
HarshdevHarshdev
361
answered Mar 27 at 23:55
HarshdevHarshdev
361
361
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55333985%2fforward-onpremise-logs-to-kinesis-firehose%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
