Styjun

Big number puzzle

Are there really no countries that protect Freedom of Speech as the United States does?

How would armour (and combat) change if the fighter didn't need to actually wear it?

Boss wants me to ignore a software API license

Escape Velocity - Won't the orbital path just become larger with higher initial velocity?

Dogfights in outer space

Causal Diagrams using Wolfram?

Why aren’t there water shutoff valves for each room?

Why did IBM make the PC BIOS source code public?

Identifying My Main Water Shutoff Valve / Setup

Why won't the Republicans use a superdelegate system like the DNC in their nomination process?

Locked Room Murder!! How and who?

An equality about sin function?

What is the hottest thing in the universe?

Crippling fear of hellfire &, damnation, please help?

Why is the result of ('b'+'a'+ + 'a' + 'a').toLowerCase() 'banana'?

Lípínguapua dopo Pêpê

Telephone number in spoken words

What would it take to get a message to another star?

"Mouth-breathing" as slang for stupidity

Graphs for which a calculus student can reasonably compute the arclength

Does fossil fuels use since 1990 account for half of all the fossil fuels used in history?

How do I call a 6-digit Australian phone number with a US-based mobile phone?

Shifting tenses in the middle of narration

Get source/upstream connection's processor name in nifi

NiFi fetchFile processor doesn't allow dynamic attributesAccessing FlowFile content in NIFI PutS3Object Processornifi error flow not working unless two files in queueMaking log files in NiFiNifi Consume AMQP parameterise queue nameHandlehttp processor in nifiPython Script using ExecuteStreamCommandNaming splitted files incrementally in nifi for a particular table and then reset for another tableDifference Between Processor Properties and Flowfile Attributes in Apache NiFiWhy is my NiFi PublishKafka processor only working with previous versions?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I want to monitor the flowfiles in Nifi from business perspective.

So I have added executescript processor using python script which creates the message and pushes the same in elasticsearch after each processor.

I want the parent processor name or id of this executescript processor so that I will keep appending in the flowfile which will allow to know through which stages/processors this flow file is passed through and I can monitor it in ELK.

rest apache-nifi

share|improve this question

edited Mar 27 at 10:52

happy

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  NiFi derives from the Flow Based Programming paradigm where each component is a "black box" that is intentionally not aware of predecessor or successor components. I would caution that trying to determine these component identities is unsupported and brittle if possible programmatically, and definitely an anti-pattern. You should re-evaluate the monitoring plan you have in mind.
  
  – Andy
  Mar 27 at 17:49
  

  
  
  
  

add a comment | 

1

I want to monitor the flowfiles in Nifi from business perspective.

So I have added executescript processor using python script which creates the message and pushes the same in elasticsearch after each processor.

I want the parent processor name or id of this executescript processor so that I will keep appending in the flowfile which will allow to know through which stages/processors this flow file is passed through and I can monitor it in ELK.

rest apache-nifi

share|improve this question

edited Mar 27 at 10:52

happy

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  NiFi derives from the Flow Based Programming paradigm where each component is a "black box" that is intentionally not aware of predecessor or successor components. I would caution that trying to determine these component identities is unsupported and brittle if possible programmatically, and definitely an anti-pattern. You should re-evaluate the monitoring plan you have in mind.
  
  – Andy
  Mar 27 at 17:49
  

  
  
  
  

add a comment | 

I want to monitor the flowfiles in Nifi from business perspective.

So I have added executescript processor using python script which creates the message and pushes the same in elasticsearch after each processor.

I want the parent processor name or id of this executescript processor so that I will keep appending in the flowfile which will allow to know through which stages/processors this flow file is passed through and I can monitor it in ELK.

rest apache-nifi

share|improve this question

edited Mar 27 at 10:52

happy

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

share|improve this question

edited Mar 27 at 10:52

happy

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

share|improve this question

edited Mar 27 at 10:52

happy

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

asked Mar 27 at 10:30

happyhappy

8951515 gold badges4545 silver badges8383 bronze badges

1 Answer
1

active

oldest

votes

2

I think the best way to monitor FlowFiles is to use the Provenance logs. You can also export those logs to ELK using another NiFi instance and S2S.

Anyway, if you want to get the processor's name of a connection's source/destination using the REST API you can get it when you browse a process group's connections.
Example:

/nifi-api/process-groups/processGroupId/connections/

You'll get an array of connections. In the connection object, you will get the name of the source in the path component/source/name. The same goes for destination.

EDIT:

To use provenance logs you need to do as follows:

1. Send provenance logs to another NiFi instance(it is restricted to NiFi since it uses S2S).


2. Parse the logs in this NiFi instance


3. Send the logs to ElasticSearch using the PutElasticSearch5 processor.

It works the best and will help you monitor the FlowFiles the best :)

share|improve this answer

edited Mar 27 at 13:24

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Is there a way to fetch it from session (session.get()) object which I get in python script?. I will explore S2S option.
  
  – happy
  Mar 27 at 11:18
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can't get it from the ProcessSession nor from ProcessContext... As I said, the best way to monitor FlowFiles is to use the Provenance Logs. You can use SiteToSiteProvenanceReportingTask to transfer the provenance logs to the other instance. It's easy too! You just need to enable site to site protocol in the destination instance(with nifi.properties)
  
  – Ben Yaakobi
  Mar 27 at 12:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I tried S2S. There I am unsure what will be Input Port name as I am not sending this to another nifi instance but sending it to elasticsearch
  
  – happy
  Mar 27 at 13:15
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It throws Error running task SiteToSiteProvenanceReportingTask[id=bf4858b6-0169-1000-b06f-640d8cba6a3e] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Provenance Events to destination due to IOException:response code 405:Method Not Allowed with explanation: null
  
  – happy
  Mar 27 at 13:16
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  check the edit I've added to my answer
  
  – Ben Yaakobi
  Mar 27 at 13:24
  

  
  
  
  

 | 
show 5 more comments

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55375008%2fget-source-upstream-connections-processor-name-in-nifi%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

I think the best way to monitor FlowFiles is to use the Provenance logs. You can also export those logs to ELK using another NiFi instance and S2S.

Anyway, if you want to get the processor's name of a connection's source/destination using the REST API you can get it when you browse a process group's connections.
Example:

/nifi-api/process-groups/processGroupId/connections/

You'll get an array of connections. In the connection object, you will get the name of the source in the path component/source/name. The same goes for destination.

EDIT:

To use provenance logs you need to do as follows:

1. Send provenance logs to another NiFi instance(it is restricted to NiFi since it uses S2S).


2. Parse the logs in this NiFi instance


3. Send the logs to ElasticSearch using the PutElasticSearch5 processor.

It works the best and will help you monitor the FlowFiles the best :)

share|improve this answer

edited Mar 27 at 13:24

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Is there a way to fetch it from session (session.get()) object which I get in python script?. I will explore S2S option.
  
  – happy
  Mar 27 at 11:18
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can't get it from the ProcessSession nor from ProcessContext... As I said, the best way to monitor FlowFiles is to use the Provenance Logs. You can use SiteToSiteProvenanceReportingTask to transfer the provenance logs to the other instance. It's easy too! You just need to enable site to site protocol in the destination instance(with nifi.properties)
  
  – Ben Yaakobi
  Mar 27 at 12:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I tried S2S. There I am unsure what will be Input Port name as I am not sending this to another nifi instance but sending it to elasticsearch
  
  – happy
  Mar 27 at 13:15
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It throws Error running task SiteToSiteProvenanceReportingTask[id=bf4858b6-0169-1000-b06f-640d8cba6a3e] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Provenance Events to destination due to IOException:response code 405:Method Not Allowed with explanation: null
  
  – happy
  Mar 27 at 13:16
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  check the edit I've added to my answer
  
  – Ben Yaakobi
  Mar 27 at 13:24
  

  
  
  
  

 | 
show 5 more comments

2

I think the best way to monitor FlowFiles is to use the Provenance logs. You can also export those logs to ELK using another NiFi instance and S2S.

Anyway, if you want to get the processor's name of a connection's source/destination using the REST API you can get it when you browse a process group's connections.
Example:

/nifi-api/process-groups/processGroupId/connections/

You'll get an array of connections. In the connection object, you will get the name of the source in the path component/source/name. The same goes for destination.

EDIT:

To use provenance logs you need to do as follows:

1. Send provenance logs to another NiFi instance(it is restricted to NiFi since it uses S2S).


2. Parse the logs in this NiFi instance


3. Send the logs to ElasticSearch using the PutElasticSearch5 processor.

It works the best and will help you monitor the FlowFiles the best :)

share|improve this answer

edited Mar 27 at 13:24

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Is there a way to fetch it from session (session.get()) object which I get in python script?. I will explore S2S option.
  
  – happy
  Mar 27 at 11:18
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can't get it from the ProcessSession nor from ProcessContext... As I said, the best way to monitor FlowFiles is to use the Provenance Logs. You can use SiteToSiteProvenanceReportingTask to transfer the provenance logs to the other instance. It's easy too! You just need to enable site to site protocol in the destination instance(with nifi.properties)
  
  – Ben Yaakobi
  Mar 27 at 12:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I tried S2S. There I am unsure what will be Input Port name as I am not sending this to another nifi instance but sending it to elasticsearch
  
  – happy
  Mar 27 at 13:15
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It throws Error running task SiteToSiteProvenanceReportingTask[id=bf4858b6-0169-1000-b06f-640d8cba6a3e] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Provenance Events to destination due to IOException:response code 405:Method Not Allowed with explanation: null
  
  – happy
  Mar 27 at 13:16
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  check the edit I've added to my answer
  
  – Ben Yaakobi
  Mar 27 at 13:24
  

  
  
  
  

 | 
show 5 more comments

I think the best way to monitor FlowFiles is to use the Provenance logs. You can also export those logs to ELK using another NiFi instance and S2S.

Anyway, if you want to get the processor's name of a connection's source/destination using the REST API you can get it when you browse a process group's connections.
Example:

/nifi-api/process-groups/processGroupId/connections/

You'll get an array of connections. In the connection object, you will get the name of the source in the path component/source/name. The same goes for destination.

EDIT:

To use provenance logs you need to do as follows:

1. Send provenance logs to another NiFi instance(it is restricted to NiFi since it uses S2S).


2. Parse the logs in this NiFi instance


3. Send the logs to ElasticSearch using the PutElasticSearch5 processor.

It works the best and will help you monitor the FlowFiles the best :)

share|improve this answer

edited Mar 27 at 13:24

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

/nifi-api/process-groups/processGroupId/connections/

share|improve this answer

edited Mar 27 at 13:24

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges

answered Mar 27 at 10:55

Ben YaakobiBen Yaakobi

6071212 bronze badges