Apache NiFi 1.1.0 – Secured cluster setup

Apache NiFi 1.1.0 is now out, and I want to discuss a specific subject in a couple of posts: how to scale up and down a NiFi cluster without loosing data? Before going into this subject, I want to setup a 3-nodes secured cluster using the NiFi toolkit. It will be my starting point to scale up my cluster with an additional node, and then scale down my cluster.

There are already great posts describing how to setup a secured cluster using embedded ZK and taking advantage of the NiFi toolkit for the certificates, so I won’t go in too much details. For reference, here are some great articles I recommend:

OK… let’s move on. My initial setup is the following:

  • My OS X laptop
  • 2 CentOS 7 VM

All nodes have required prerequisites (network, java, etc), have the NiFi 1.1.0 binary archive uncompressed available, and have the NiFi toolkit 1.1.0 binary archive uncompressed available.

On my OS X laptop, I will use the NiFi TLS toolkit in server mode so that it acts as a Certificate Authority that can be used by clients to get Certificates.

Here is the description of how to use the TLS toolkit in server mode:

./bin/tls-toolkit.sh server

usage: org.apache.nifi.toolkit.tls.TlsToolkitMain [-a <arg>] [-c <arg>] [--configJsonIn <arg>] [-d <arg>] [-D <arg>] [-f <arg>] [-F] [-g] [-h] [-k <arg>] [-p
       <arg>] [-s <arg>] [-T <arg>] [-t <arg>]

Acts as a Certificate Authority that can be used by clients to get Certificates

 -a,--keyAlgorithm <arg>                   Algorithm to use for generated keys. (default: RSA)
 -c,--certificateAuthorityHostname <arg>   Hostname of NiFi Certificate Authority (default: localhost)
    --configJsonIn <arg>                   The place to read configuration info from (defaults to the value of configJson), implies useConfigJson if set.
                                           (default: configJson value)
 -d,--days <arg>                           Number of days issued certificate should be valid for. (default: 1095)
 -D,--dn <arg>                             The dn to use for the CA certificate (default: CN=YOUR_CA_HOSTNAME,OU=NIFI)
 -f,--configJson <arg>                     The place to write configuration info (default: config.json)
 -F,--useConfigJson                        Flag specifying that all configuration is read from configJson to facilitate automated use (otherwise configJson will
                                           only be written to.
 -g,--differentKeyAndKeystorePasswords     Use different generated password for the key and the keyStore.
 -h,--help                                 Print help and exit.
 -k,--keySize <arg>                        Number of bits for generated keys. (default: 2048)
 -p,--PORT <arg>                           The port for the Certificate Authority to listen on (default: 8443)
 -s,--signingAlgorithm <arg>               Algorithm to use for signing certificates. (default: SHA256WITHRSA)
 -T,--keyStoreType <arg>                   The type of keyStores to generate. (default: jks)
 -t,--token <arg>                          The token to use to prevent MITM (required and must be same as one used by clients)

In my case, I run the TLS toolkit on my node-3, and I run the following command:

./bin/tls-toolkit.sh server -c node-3 -t myTokenToUseToPreventMITM -p 9999

On each of my node, I created a nifi directory in /etc and I ran a command using the toolkit to get my signed certificates generated into my current directory:

.../bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999

And I have the following generated files on each of my nodes:

  • config.json
  • keystore.jks
  • nifi-cert.pem
  • truststore.jks

I now configure ZooKeeper as described here. Here is a short list of the tasks (in bold what slightly changed in comparison with my previous post):

  • configure conf/zookeeper.properties to list the ZK nodes
  • configure ZK state ID file
  • set nifi.state.management.embedded.zookeeper.start
  • set nifi.zookeeper.connect.string
  • set nifi.cluster.protocol.is.secure=true
  • set nifi.cluster.is.node=true
  • set nifi.cluster.node.address=node-<1-3>
  • set nifi.cluster.node.protocol.port=9998
  • set nifi.remote.input.host=node-<1-3>
  • set nifi.remote.input.secure=true
  • set nifi.remote.input.socket.port=9997
  • set nifi.web.https.host=node-<1-3>
  • set nifi.web.https.port=8443

(ports are changed to ensure there is no conflict with the TLS toolkit running the CA server)

I now configure the following properties with what has been generated by the toolkit:


For one of my node, it gives:


Then I generate a certificate for myself as a client to be able to authenticate against NiFi UI:

.../bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999 -D "CN=pvillard,OU=NIFI" -T PKCS12

Don’t forget the -T option to get your client certificate in a format that is easy to import in your browser (PKCS12). This command also generates a nifi-cert.pem file that corresponds to the CA certificate, you will need to import it in your browser as well (and you might need to manually update the trust level on this certificate to ensure you have access to the UI).

At this point I’m able to fill the authorizers.xml file. I need to specify myself as initial admin identity (to access the UI with full administration rights), and specify each nodes of my cluster (using the DN provided with the generated certificates). It gives:

        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Users File">./conf/users.xml</property>
        <property name="Initial Admin Identity">CN=pvillard, OU=NIFI</property>
        <property name="Legacy Authorized Users File"></property>

        <property name="Node Identity 1">CN=node-1, OU=NIFI</property>
        <property name="Node Identity 2">CN=node-2, OU=NIFI</property>
        <property name="Node Identity 3">CN=node-3, OU=NIFI</property>

WARNING – Please be careful when updating this file because identities are case-sensitive and blank-sensitive. For example, even though I specified

-D "CN=pvillard,OU=NIFI"

when executing the command to generate the certificates, it introduced a white space after the comma. The correct string to use in the configuration file is given in the output of the TLS toolkit when executing the command.

Once I’ve updated this file on each node, I’m now ready to start each node of the cluster.

./bin/nifi.sh start && tail -f ./logs/nifi-app.log

Once all nodes are correctly started, I am now able to access the NiFi UI using any of the nodes in the cluster:

NiFi UI / 3-nodes secured cluster
NiFi UI / 3-nodes secured cluster

That’s all! Next post will use the current environment as a starting point to demonstrate how to scale up/down a NiFi cluster.

30 thoughts on “Apache NiFi 1.1.0 – Secured cluster setup

  1. I am having issues using the pem file to authenticate in my browser. Whenever I try to upload it, it tells me “This personal certificate can’t be installed because you do not own the corresponding private key which was created when the certificate was requested.”

    I have NiFi installed on two different servers. I got my certificate signed by the CA server that is being hosted on one of the servers.

    NiFi is up and running, I just get insufficient permissions when I log in.

    How can I manipulate the pem file in order to be able to log in as the initial admin identity?


    • There are two things to add:
      – the .pem file should be the CA certificate and should be added in the CA configuration of your browser. This file will be used to trust the nodes you are requesting.
      – the .p12 file that is your private certificate that you will need to add in your browser as well. This file will be used to authenticate yourself against the node. When adding the file in the browser, (if I recall correctly) you’ll have to give a password that you should be able to find in the json file generated with the request to generate your own user certificate.


  2. Hi Pierre. I hope you could provide some guidance to us as to how to secure our NiFi (1.3.0) instance we have just installed. It is a single-node one, running on a CentOS box, but we will be connecting to it remotely from Windows desktop computers. In the un-secured mode, there are no issues.

    I followed the recommendations from http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy, specifically:
    * ran the tls-toolkit with the “standalone -n ‘localhost'” options
    * updated the respective NiFi configuration files
    * imported both .pem and .p12 files into the Chrome browser on my desktop

    But when trying to connect to NiFi from there, I get the ERR_BAD_SSL_CLIENT_AUTH_CERT response. Someone suggested using the FQDN of the NiFi server instead of ‘localhost’ when executing the tls-toolkit, which I did next. Unfortunately, that didn’t help. I wonder what might be wrong with our setup?


    • Hi Alexander, sorry for the late answer, I was away from a keyboard. Did you manager to solve your issue? Could be a SAN issue. What’s the URL you’re typing when trying to access the UI? Is the FQDN of this URL matching the SAN fields of your certificate? Starting with Chrome 53 (if I recall correctly), hostname MUST be defined as a SAN. Otherwise access will be denied.


  3. Hi Pierre,I can’t generate client certificate. I got error as below when I tried executing client certificate command:

    INFO [main] org.apache.nifi.toolkit.tls.commandLine.BaseCommandLine: Command line argument –keyStoreType=PKCS12 only applies to keystore, recommended truststore type of JKS unaffected.
    Service client error: null

    Usage: tls-toolkit service [-h] [args]

    standalone: Creates certificates and config files for nifi cluster.
    server: Acts as a Certificate Authority that can be used by clients to get Certificates
    client: Generates a private key and gets it signed by the certificate authority.


  4. Please forget about it. Just found that is some config-client.json that is configured to check in localhost 8443. Changed it to point to my toolkit server and Voila!
    Hope it helps


  5. Sorry to bother again, but finally found the problem.
    java.lang.NullPointerException: null
    at org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClientCommandLine.getDnHostname(TlsCertificateAuthorityClientCommandLine.java:106)

    So your hostname has to be resolvable. In my case ip-10-x-x-x wasn’t in my dns server so I added my ip to my own /etc/hosts and worked!!!


  6. I’m using toolkit-1.2.0 when I try and run ./bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999 I get the following error:
    Service client error: null

    Usage: tls-toolkit service [-h] [args]

    standalone: Creates certificates and config files for nifi cluster.
    server: Acts as a Certificate Authority that can be used by clients to get Certificates
    client: Generates a private key and gets it signed by the certificate authority.
    It’s as if it doesn’t recognize the command?


  7. Hi pvillard31,

    We followed the post and create a 3 node nifi cluster in AWS (Nifi 1.5), but we are getting “peer not authenticated” .

    Can you please let us know the steps to resolve the same?

    Thanks in Advance.

    Liked by 1 person

  8. Hi pvillard31,

    could you please tell me in short what are the steps to configure 3 node nifi cluster on phisically separated servers with zookeper quorum up and running. This article is great and it is for virtual nodes on the same machine, am I right?

    Sholud I generate certificates on all the servers? Also, I’m using kerberos for authenticating users in nifi.
    I’ve managed to start one node using kerberos and all it is all fine.

    I followed this article:



    • Hi Tom, it should not make any difference I believe.

      Just note that using embedded ZK is not recommended for production setup, using a ZK quorum hosted on dedicated nodes with proper disks should be preferred.

      Yes you should generate certificates for each node (don’t forget the SANs) and you can you use the toolkit to ease this process unless you want to use your own CA.



      • Ok, thanks.

        What do you mean with SANs (don’t fortget the SANs)?
        How can I do that?

        One more question, what is the difference between Standalone mode and Client/Server mode of the toolkit?

        I see you used Client/Server toolkit mode for the nodes certificates and for your own certificate to authenticate yourself for the Nifi UI.
        I authenticate myself using Kerberos realm and valid Kerberos user, but for the nodes certificate I used standalone toolkit mode.



  9. Hi Pierre, thanks for the great post! However, I try similar setup with nifi 1.8.0, always get “Untrusted proxy” error, what could go wrong?


    • Hi Liyi, did you add the proxy permissions to your nodes? That’s probably the issue. Have a look at authorizations.xml and users.xml files where your node identities should be defined.


  10. thanks for detailed steps. A couple of questions
    1. Is it mandatory to import client certification in a browser if we need to access UI
    2. Will, it also secure REST API too, any documentation, if it is not securing REST endpoint
    3. I am getting No applicable policies could be found. Contact the system administrator. I have following in my xml

    MyHost, OU=NIFI


    • 1. No – you can access the UI using certs, Kerberos, login/password (if you followed the required steps to define a LDAP, see another post of mines), or even OIDC if you want to delegate authentication.
      2. Yes REST API will also be secured
      3. Would need more details and also, check the initial admin parts.


  11. Hi i have CA server on node 3 , client cets generated on node 3
    whereas node 2 & node 1 unable to generate client certs , error
    Service client error

    does it a port communication problem?

    ./opt/nifi-toolkit-1.9.2/bin/tls-toolkit.sh client -c ip-10-xxxx.xxxx-250.xxx.xxx -t xxxx@nifi -p 8443
    2020/01/27 21:50:54 INFO [main] org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient: Requesting new certificate from ip-10-xxxx.xxxx-250.xxxx.xxxx:8443
    2020/01/27 21:50:55 INFO [main] org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPerformer: Requesting certificate with dn CN=ip-10-xxxx.xxxx-240.xxxx.xxxx,OU=NIFI from ip-10-xxxx.xxxx-250.xxxx.xxxx:8443

    Service client error: Connect to ip-10-xxxx.xxxx-250.xxxx.xxxx:8443 [ip-10-xxxx.xxxx-250.xxx.xxx/] failed: Connection timed out (Connection timed out)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.