Styjun

I am running MiNiFi in a windows server which is behind my company's firewall. My NiFi is running on Hortonworks cluster (3 nodes ex:sj46,sj47,sj48).

I am trying to send data from the MiNiFi to NiFi.

• MiNiFi version: 0.4


• NiFi version: 1.5

In MiNiFi, i have ListFile processor, FetchFile processor and RemoteProcessGroup.

RemoteProcessGroup:

• URL:
  https://sjxx46.xxx:6689/nifi,https://sjxx47.xxx:6689/nifi,https://sjxx48.xxx:6689/nifi
  


• TransportProtocol: HTTP

NiFi.properties:

nifi.remote.input.host=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.input.secure=True
nifi.remote.input.socket.port=6698

nifi.cluster.node.protocol.port=6699
nifi.web.http.port=
nifi.web.https.host=sj46.xxxx
nifi.web.https.port=6689

MiNiFi config.yml

MiNiFi Config Version: 3
Flow Controller:
 name: min2510
 comment: ''
Core Properties:
 flow controller graceful shutdown period: 10 sec
 flow service write delay interval: 500 ms
 administrative yield duration: 30 sec
 bored yield duration: 10 millis
 max concurrent threads: 1
 variable registry properties: ''
FlowFile Repository:
 partitions: 256
 checkpoint interval: 2 mins
 always sync: false
 Swap:
 threshold: 20000
 in period: 5 sec
 in threads: 1
 out period: 5 sec
 out threads: 4
Content Repository:
 content claim max appendable size: 10 MB
 content claim max flow files: 100
 always sync: false
Provenance Repository:
 provenance rollover time: 1 min
 implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
 buffer size: 1440
 snapshot frequency: 1 min
Security Properties:
 keystore: ''
 keystore type: ''
 keystore password: ''
 key password: ''
 truststore: ''
 truststore type: ''
 truststore password: ''
 ssl protocol: ''
 Sensitive Props:
 key:
 algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
 provider: BC
Processors:
- id: 50420aac-ef66-30ac-0000-000000000000
 name: FetchFile
 class: org.apache.nifi.processors.standard.FetchFile
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 - failure
 - not.found
 - permission.denied
 Properties:
 Completion Strategy: None
 File to Fetch: $absolute.path/$filename
 Log level when file not found: ERROR
 Log level when permission denied: ERROR
 Move Conflict Strategy: Rename
 Move Destination Directory:
- id: ae4799d5-073c-3dd6-0000-000000000000
 name: ListFile
 class: org.apache.nifi.processors.standard.ListFile
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list: []
 Properties:
 File Filter: '[^.].*'
 Ignore Hidden Files: 'true'
 Include File Attributes: 'true'
 Input Directory: C:minifi to EFT
 Input Directory Location: Local
 Maximum File Age:
 Maximum File Size:
 Minimum File Age: 0 sec
 Minimum File Size: 0 B
 Path Filter:
 Recurse Subdirectories: 'true'
 target-system-timestamp-precision: auto-detect
Controller Services: []
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 95dde9e7-96a4-3bea-0000-000000000000
 name: FetchFile/success/eef78a5f-cc55-1d33-0000-0000481921e3
 source id: 50420aac-ef66-30ac-0000-000000000000
 source relationship names:
 - success
 destination id: eef78a5f-cc55-1d33-0000-0000481921e3
 max work queue size: 10000
 max work queue data size: 1 GB
 flowfile expiration: 0 sec
 queue prioritizer class: ''
- id: 2ba13196-5ac6-3966-0000-000000000000
 name: ListFile/success/FetchFile
 source id: ae4799d5-073c-3dd6-0000-000000000000
 source relationship names:
 - success
 destination id: 50420aac-ef66-30ac-0000-000000000000
 max work queue size: 10000
 max work queue data size: 1 GB
 flowfile expiration: 0 sec
 queue prioritizer class: ''
Remote Process Groups:
- id: 6bd4b280-646b-3b93-0000-000000000000
 name: ''
 url: https://sj46.xxxx:6689/nifi
 comment: ''
 timeout: 30 sec
 yield period: 10 sec
 transport protocol: HTTP
 proxy host: ''
 proxy port: ''
 proxy user: ''
 proxy password: ''
 local network interface: ''
 Input Ports:
 - id: eef78a5f-cc55-1d33-0000-0000481921e3
 name: From MiNiFi
 comment: ''
 max concurrent tasks: 1
 use compression: false
 Output Ports: []
NiFi Properties Overrides: 

in the MiNiFi windows server, the ports such as 6688,6689 and 6699 are opened.

but if i try to telnet on MiNiFi and NiFi, it was not established.

minifi-app.log

2019-03-27 15:50:09,152 WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://sj46.xxxx:6689/nifi-api due to org.apache.http.conn.HttpHostConnectException: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
2019-03-27 15:50:09,160 WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to communicate with remote instance RemoteProcessGroup[https://sj46.xxxx:6689/nifi]
org.apache.nifi.controller.exception.CommunicationsException: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://sj46.xxxx:6689/nifi due to: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
 at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:882)
 at org.apache.nifi.remote.StandardRemoteProcessGroup.lambda$initialize$0(StandardRemoteProcessGroup.java:193)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://sj46.xxxx:6689/nifi due to: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
 at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:844)
 ... 8 common frames omitted

Any pointers on how to troubleshoot this issue?

the above issue got resolved by disabling the firewall from MiNiFi server.
now i am getting another issue like below:

WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://sj46.xxxx:6689/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to connect to RemoteProcessGroup[https://sj46.xxxx:6689/nifi] due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

the above issue of SSLHandshakeException has been resolved by importing the SSl certificate (which you can refer in NiFi.properties especially keystore.jks and truststore.jks) to the MiNiFi. we need to manually modify config.yml in MiNiFi.

after this issue, i was facing the below one:

ERROR [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunConnectableTask RemoteGroupPort[name=From MiNiFi,targets=https://sj46.xxxx:6689/nifi] failed to process session due to java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;

ERROR [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunConnectableTask 
java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;

the SSLContext issue has been resolved by changing the MiNiFi version from 0.4 to 0.3.

Now i am successfully able to fetch data from MiNiFi server to the NiFi.

thanks.

I see that your remote port is configured at 6698.

1. Try to telnet the NiFi Host on 6698 from the machine where MiNiFi is running. Try to see the inbound rules configured on the NiFi Host. (Telnetting to a remote host on a specific port is to check whether you are able to communicate with that host over that specific port. It has got nothing to do with whether the specific port on your machine is open or not. )



2. 
   

   Add the NiFi Host name in nifi.properties under #Site to Site Properties

   
   
   

   nifi.remote.input.host=Host where NiFi is running

   
   


3. 
   

   I would suggest, just to remove the mist of confusion over the common patterns of NiFi Web URL Port and Remote Input Port, try changing this in nifi.properties

   
   
   

   nifi.remote.input.socket.port=1026

   
   


4. 
   

   Also add the below section under Input Ports in config.yml:

   
   
   

   Properties:
   
   Port: 6698
   Host Name:

   
   

For config.yml and nifi.properties modification references, check this.