Apache NiFi 1.1.0 is now out, and I want to discuss a specific subject in a couple of posts: how to scale up and down a NiFi cluster without loosing data? Before going into this subject, I want to setup a 3-nodes secured cluster using the NiFi toolkit. It will be my starting point to scale up my cluster with an additional node, and then scale down my cluster.
There are already great posts describing how to setup a secured cluster using embedded ZK and taking advantage of the NiFi toolkit for the certificates, so I won’t go in too much details. For reference, here are some great articles I recommend:
- Apache NiFi 1.0.0 – Authorization and Multi-Tenancy
- Apache NiFi tls-toolkit multi-node standalone in Docker
- Using the TLS toolkit to simplify security
OK… let’s move on. My initial setup is the following:
- My OS X laptop
- 2 CentOS 7 VM
All nodes have required prerequisites (network, java, etc), have the NiFi 1.1.0 binary archive uncompressed available, and have the NiFi toolkit 1.1.0 binary archive uncompressed available.
On my OS X laptop, I will use the NiFi TLS toolkit in server mode so that it acts as a Certificate Authority that can be used by clients to get Certificates.
Here is the description of how to use the TLS toolkit in server mode:
./bin/tls-toolkit.sh server usage: org.apache.nifi.toolkit.tls.TlsToolkitMain [-a <arg>] [-c <arg>] [--configJsonIn <arg>] [-d <arg>] [-D <arg>] [-f <arg>] [-F] [-g] [-h] [-k <arg>] [-p <arg>] [-s <arg>] [-T <arg>] [-t <arg>] Acts as a Certificate Authority that can be used by clients to get Certificates -a,--keyAlgorithm <arg> Algorithm to use for generated keys. (default: RSA) -c,--certificateAuthorityHostname <arg> Hostname of NiFi Certificate Authority (default: localhost) --configJsonIn <arg> The place to read configuration info from (defaults to the value of configJson), implies useConfigJson if set. (default: configJson value) -d,--days <arg> Number of days issued certificate should be valid for. (default: 1095) -D,--dn <arg> The dn to use for the CA certificate (default: CN=YOUR_CA_HOSTNAME,OU=NIFI) -f,--configJson <arg> The place to write configuration info (default: config.json) -F,--useConfigJson Flag specifying that all configuration is read from configJson to facilitate automated use (otherwise configJson will only be written to. -g,--differentKeyAndKeystorePasswords Use different generated password for the key and the keyStore. -h,--help Print help and exit. -k,--keySize <arg> Number of bits for generated keys. (default: 2048) -p,--PORT <arg> The port for the Certificate Authority to listen on (default: 8443) -s,--signingAlgorithm <arg> Algorithm to use for signing certificates. (default: SHA256WITHRSA) -T,--keyStoreType <arg> The type of keyStores to generate. (default: jks) -t,--token <arg> The token to use to prevent MITM (required and must be same as one used by clients)
In my case, I run the TLS toolkit on my node-3, and I run the following command:
./bin/tls-toolkit.sh server -c node-3 -t myTokenToUseToPreventMITM -p 9999
On each of my node, I created a nifi directory in /etc and I ran a command using the toolkit to get my signed certificates generated into my current directory:
.../bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999
And I have the following generated files on each of my nodes:
- config.json
- keystore.jks
- nifi-cert.pem
- truststore.jks
I now configure ZooKeeper as described here. Here is a short list of the tasks (in bold what slightly changed in comparison with my previous post):
- configure conf/zookeeper.properties to list the ZK nodes
- configure ZK state ID file
- set nifi.state.management.embedded.zookeeper.start
- set nifi.zookeeper.connect.string
- set nifi.cluster.protocol.is.secure=true
- set nifi.cluster.is.node=true
- set nifi.cluster.node.address=node-<1-3>
- set nifi.cluster.node.protocol.port=9998
- set nifi.remote.input.host=node-<1-3>
- set nifi.remote.input.secure=true
- set nifi.remote.input.socket.port=9997
- set nifi.web.https.host=node-<1-3>
- set nifi.web.https.port=8443
(ports are changed to ensure there is no conflict with the TLS toolkit running the CA server)
I now configure the following properties with what has been generated by the toolkit:
nifi.security.keystore= nifi.security.keystoreType= nifi.security.keystorePasswd= nifi.security.keyPasswd= nifi.security.truststore= nifi.security.truststoreType= nifi.security.truststorePasswd= nifi.security.needClientAuth=
For one of my node, it gives:
nifi.security.keystore=/etc/nifi/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=4nVzULZ+CBiPCePvFSCw4LDAvzNoumAqu+TcyeDQ1ac nifi.security.keyPasswd=4nVzULZ+CBiPCePvFSCw4LDAvzNoumAqu+TcyeDQ1ac nifi.security.truststore=/etc/nifi/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=0fFJ+pd4qkso0jC0jh7w7tLPPRSINYI6of+KnRBRVSw nifi.security.needClientAuth=true
Then I generate a certificate for myself as a client to be able to authenticate against NiFi UI:
.../bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999 -D "CN=pvillard,OU=NIFI" -T PKCS12
Don’t forget the -T option to get your client certificate in a format that is easy to import in your browser (PKCS12). This command also generates a nifi-cert.pem file that corresponds to the CA certificate, you will need to import it in your browser as well (and you might need to manually update the trust level on this certificate to ensure you have access to the UI).
At this point I’m able to fill the authorizers.xml file. I need to specify myself as initial admin identity (to access the UI with full administration rights), and specify each nodes of my cluster (using the DN provided with the generated certificates). It gives:
<authorizers> <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">CN=pvillard, OU=NIFI</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=node-1, OU=NIFI</property> <property name="Node Identity 2">CN=node-2, OU=NIFI</property> <property name="Node Identity 3">CN=node-3, OU=NIFI</property> </authorizer> </authorizers>
WARNING – Please be careful when updating this file because identities are case-sensitive and blank-sensitive. For example, even though I specified
-D "CN=pvillard,OU=NIFI"when executing the command to generate the certificates, it introduced a white space after the comma. The correct string to use in the configuration file is given in the output of the TLS toolkit when executing the command.
Once I’ve updated this file on each node, I’m now ready to start each node of the cluster.
./bin/nifi.sh start && tail -f ./logs/nifi-app.log
Once all nodes are correctly started, I am now able to access the NiFi UI using any of the nodes in the cluster:
That’s all! Next post will use the current environment as a starting point to demonstrate how to scale up/down a NiFi cluster.
Pierre Villard 
[…] by adding a new node? Here is a way to do it without restarting the cluster. This post starts with a 3-nodes secured cluster with embedded ZooKeeper and we are going to add a new node to this cluster (and this new node won’t have an embedded […]
LikeLike
[…] Apache NiFi 1.1.0 – Secured cluster setup by Pierre Villard […]
LikeLike
[…] need to add this user in the list of users through NiFi UI using the initial admin account (see Apache NiFi 1.1.0 – Secured cluster setup). At there is no syncing mechanism to automatically add LDAP users/groups into […]
LikeLike
[…] assume I already have my secured NiFi 3-nodes cluster up and running. And also, I’ve already setup my Docker and Portainer environment: I’m ready to pull […]
LikeLike
Hi Thanks ,
I have followed your steps to configure 3 node Cluster setup but i am getting the below error.
https://community.hortonworks.com/questions/92375/secured-nifi-cluster-setup.html
LikeLike
[…] nodes for NiFi – secured using the NiFi CA (provided with the NiFi […]
LikeLike
I am having issues using the pem file to authenticate in my browser. Whenever I try to upload it, it tells me “This personal certificate can’t be installed because you do not own the corresponding private key which was created when the certificate was requested.”
I have NiFi installed on two different servers. I got my certificate signed by the CA server that is being hosted on one of the servers.
NiFi is up and running, I just get insufficient permissions when I log in.
How can I manipulate the pem file in order to be able to log in as the initial admin identity?
LikeLike
There are two things to add:
– the .pem file should be the CA certificate and should be added in the CA configuration of your browser. This file will be used to trust the nodes you are requesting.
– the .p12 file that is your private certificate that you will need to add in your browser as well. This file will be used to authenticate yourself against the node. When adding the file in the browser, (if I recall correctly) you’ll have to give a password that you should be able to find in the json file generated with the request to generate your own user certificate.
LikeLike
Hi Pierre. I hope you could provide some guidance to us as to how to secure our NiFi (1.3.0) instance we have just installed. It is a single-node one, running on a CentOS box, but we will be connecting to it remotely from Windows desktop computers. In the un-secured mode, there are no issues.
I followed the recommendations from http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy, specifically:
* ran the tls-toolkit with the “standalone -n ‘localhost'” options
* updated the respective NiFi configuration files
* imported both .pem and .p12 files into the Chrome browser on my desktop
But when trying to connect to NiFi from there, I get the ERR_BAD_SSL_CLIENT_AUTH_CERT response. Someone suggested using the FQDN of the NiFi server instead of ‘localhost’ when executing the tls-toolkit, which I did next. Unfortunately, that didn’t help. I wonder what might be wrong with our setup?
LikeLike
Hi Alexander, sorry for the late answer, I was away from a keyboard. Did you manager to solve your issue? Could be a SAN issue. What’s the URL you’re typing when trying to access the UI? Is the FQDN of this URL matching the SAN fields of your certificate? Starting with Chrome 53 (if I recall correctly), hostname MUST be defined as a SAN. Otherwise access will be denied.
LikeLike
Hi Pierre,I can’t generate client certificate. I got error as below when I tried executing client certificate command:
INFO [main] org.apache.nifi.toolkit.tls.commandLine.BaseCommandLine: Command line argument –keyStoreType=PKCS12 only applies to keystore, recommended truststore type of JKS unaffected.
Service client error: null
Usage: tls-toolkit service [-h] [args]
Services:
standalone: Creates certificates and config files for nifi cluster.
server: Acts as a Certificate Authority that can be used by clients to get Certificates
client: Generates a private key and gets it signed by the certificate authority.
LikeLike
Hi, what command are you executing exactly?
LikeLike
nifi-toolkit-1.3.0/bin/tls-toolkit.sh client -h
tls-toolkit.sh: JAVA_HOME not set; results may vary
Service client error: null
LikeLike
Please forget about it. Just found that is some config-client.json that is configured to check in localhost 8443. Changed it to point to my toolkit server and Voila!
Hope it helps
LikeLike
…. not really. Got it wrong was trying tls-toolkit.sh server . Not client 😦
Still same error
LikeLike
Sorry to bother again, but finally found the problem.
java.lang.NullPointerException: null
at org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClientCommandLine.getDnHostname(TlsCertificateAuthorityClientCommandLine.java:106)
So your hostname has to be resolvable. In my case ip-10-x-x-x wasn’t in my dns server so I added my ip to my own /etc/hosts and worked!!!
LikeLike
Thanks for pointing it Francisco, that’s something that we should improve to make it clear.
LikeLike
Glad to help. Anytime 🙂
LikeLike
I’m using toolkit-1.2.0 when I try and run ./bin/tls-toolkit.sh client -c node-3 -t myTokenToUseToPreventMITM -p 9999 I get the following error:
Service client error: null
Usage: tls-toolkit service [-h] [args]
Services:
standalone: Creates certificates and config files for nifi cluster.
server: Acts as a Certificate Authority that can be used by clients to get Certificates
client: Generates a private key and gets it signed by the certificate authority.
It’s as if it doesn’t recognize the command?
LikeLike
Hi pvillard31,
We followed the post and create a 3 node nifi cluster in AWS (Nifi 1.5), but we are getting “peer not authenticated” .
Can you please let us know the steps to resolve the same?
Thanks in Advance.
Santhosh
LikeLiked by 1 person
Hi Santhosh, I’d need more details about the errors you’re getting and how you configured the instances.
LikeLiked by 1 person
Hi pvillard31,
could you please tell me in short what are the steps to configure 3 node nifi cluster on phisically separated servers with zookeper quorum up and running. This article is great and it is for virtual nodes on the same machine, am I right?
Sholud I generate certificates on all the servers? Also, I’m using kerberos for authenticating users in nifi.
I’ve managed to start one node using kerberos and all it is all fine.
I followed this article:
https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/
Thanks,
Tom
LikeLike
Hi Tom, it should not make any difference I believe.
Just note that using embedded ZK is not recommended for production setup, using a ZK quorum hosted on dedicated nodes with proper disks should be preferred.
Yes you should generate certificates for each node (don’t forget the SANs) and you can you use the toolkit to ease this process unless you want to use your own CA.
Pierre
LikeLike
Ok, thanks.
What do you mean with SANs (don’t fortget the SANs)?
How can I do that?
One more question, what is the difference between Standalone mode and Client/Server mode of the toolkit?
I see you used Client/Server toolkit mode for the nodes certificates and for your own certificate to authenticate yourself for the Nifi UI.
I authenticate myself using Kerberos realm and valid Kerberos user, but for the nodes certificate I used standalone toolkit mode.
Thanks,
Tom
LikeLike
If you want to use a VIP or to avoid any issue when accessing NiFi with your browser when secured, certificates should contain SAN(s) (Subjective Alternative Names). The toolkit takes care of it if you use it to generate certificates for the nodes.
Have a look here (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit) for the toolkit documentation. Let me know if something is not clear.
LikeLike
Hi Pierre, thanks for the great post! However, I try similar setup with nifi 1.8.0, always get “Untrusted proxy” error, what could go wrong?
LikeLike
Hi Liyi, did you add the proxy permissions to your nodes? That’s probably the issue. Have a look at authorizations.xml and users.xml files where your node identities should be defined.
LikeLike
thanks for detailed steps. A couple of questions
1. Is it mandatory to import client certification in a browser if we need to access UI
2. Will, it also secure REST API too, any documentation, if it is not securing REST endpoint
3. I am getting No applicable policies could be found. Contact the system administrator. I have following in my xml
file-provider
org.apache.nifi.authorization.FileAuthorizer
./conf/authorizations.xml
./conf/users.xml
MyHost, OU=NIFI
LikeLike
1. No – you can access the UI using certs, Kerberos, login/password (if you followed the required steps to define a LDAP, see another post of mines), or even OIDC if you want to delegate authentication.
2. Yes REST API will also be secured
3. Would need more details and also, check the initial admin parts.
LikeLike
Hi i have CA server on node 3 , client cets generated on node 3
whereas node 2 & node 1 unable to generate client certs , error
Service client error
does it a port communication problem?
./opt/nifi-toolkit-1.9.2/bin/tls-toolkit.sh client -c ip-10-xxxx.xxxx-250.xxx.xxx -t xxxx@nifi -p 8443
2020/01/27 21:50:54 INFO [main] org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient: Requesting new certificate from ip-10-xxxx.xxxx-250.xxxx.xxxx:8443
2020/01/27 21:50:55 INFO [main] org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPerformer: Requesting certificate with dn CN=ip-10-xxxx.xxxx-240.xxxx.xxxx,OU=NIFI from ip-10-xxxx.xxxx-250.xxxx.xxxx:8443
Service client error: Connect to ip-10-xxxx.xxxx-250.xxxx.xxxx:8443 [ip-10-xxxx.xxxx-250.xxx.xxx/10.175.124.250] failed: Connection timed out (Connection timed out)
LikeLike