Integration of NiFi with LDAP

Once your cluster is secured, you probably want to start allowing users to access the cluster and you may not want to issue individual certificates for each user. In this case, one of the option is to use LDAP as the authentication provider of NiFi. This is quite simple, and we’ll see in this post how to easily setup a local LDAP server and integrate NiFi with it.

In terms of configuration, everything is done with two files:

  • ./conf/nifi.properties
  • ./conf/login-identity-providers.xml

In nifi.properties, we are interested by two properties:

nifi.login.identity.provider.configuration.file
nifi.security.user.login.identity.provider

The first one is used to give the path to the login-identity-providers.xml and the second one is used to define the name of the identity provider to use from the XML file (in case you configured multiple providers).

A quick quote from the documentation:

NiFi supports user authentication via client certificates or via username/password. Username/password authentication is performed by a Login Identity Provider. The Login Identity Provider is a pluggable mechanism for authenticating users via their username/password. Which Login Identity Provider to use is configured in two properties in the nifi.properties file.

The nifi.login.identity.provider.configuration.file property specifies the configuration file for Login Identity Providers. The nifi.security.user.login.identity.provider property indicates which of the configured Login Identity Provider should be used. If this property is not configured, NiFi will not support username/password authentication and will require client certificates for authenticating users over HTTPS. By default, this property is not configured meaning that username/password must be explicitly enabled.

NiFi does not perform user authentication over HTTP. Using HTTP all users will be granted all roles.

In other words, if you want login/password authentication, your cluster needs to be secured first!

OK, so I set the following values in nifi.properties:

nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.security.user.login.identity.provider=ldap-provider

And then I just need to configure my XML files and to restart NiFi. Here are the LDAP parameters (and we can notice that the identifier is matching the value set in nifi.properties):

<provider>
        <identifier>ldap-provider</identifier>
        <class>org.apache.nifi.ldap.LdapProvider</class>
        <property name="Authentication Strategy">START_TLS</property>
        <property name="Manager DN"></property>
        <property name="Manager Password"></property>
        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>
        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>
        <property name="Url"></property>
        <property name="User Search Base"></property>
        <property name="User Search Filter"></property>
        <property name="Identity Strategy">USE_DN</property>
        <property name="Authentication Expiration">12 hours</property>
    </provider>

And here is the associated documentation:

Identity Provider for users logging in with username/password against an LDAP server.

‘Authentication Strategy’ – How the connection to the LDAP server is authenticated. Possible values are ANONYMOUS, SIMPLE, LDAPS, or START_TLS.

‘Manager DN’ – The DN of the manager that is used to bind to the LDAP server to search for users.
‘Manager Password’ – The password of the manager that is used to bind to the LDAP server to search for users.

‘TLS – Keystore’ – Path to the Keystore that is used when connecting to LDAP using LDAPS or START_TLS.
‘TLS – Keystore Password’ – Password for the Keystore that is used when connecting to LDAP using LDAPS or START_TLS.
‘TLS – Keystore Type’ – Type of the Keystore that is used when connecting to LDAP using LDAPS or START_TLS (i.e. JKS or PKCS12).
‘TLS – Truststore’ – Path to the Truststore that is used when connecting to LDAP using LDAPS or START_TLS.
‘TLS – Truststore Password’ – Password for the Truststore that is used when connecting to LDAP using LDAPS or START_TLS.
‘TLS – Truststore Type’ – Type of the Truststore that is used when connecting to LDAP using LDAPS or START_TLS (i.e. JKS or PKCS12).
‘TLS – Client Auth’ – Client authentication policy when connecting to LDAP using LDAPS or START_TLS. Possible values are REQUIRED, WANT, NONE.
‘TLS – Protocol’ – Protocol to use when connecting to LDAP using LDAPS or START_TLS. (i.e. TLS, TLSv1.1, TLSv1.2, etc).
‘TLS – Shutdown Gracefully’ – Specifies whether the TLS should be shut down gracefully before the target context is closed. Defaults to false.

‘Referral Strategy’ – Strategy for handling referrals. Possible values are FOLLOW, IGNORE, THROW.
‘Connect Timeout’ – Duration of connect timeout. (i.e. 10 secs).
‘Read Timeout’ – Duration of read timeout. (i.e. 10 secs).

‘Url’ – Url of the LDAP server (i.e. ldap://<hostname>:<port>).
User Search Base’ – Base DN for searching for users (i.e. CN=Users,DC=example,DC=com).
User Search Filter’ – Filter for searching for users against the ‘User Search Base’. (i.e. sAMAccountName={0}). The user specified name is inserted into ‘{0}’.

‘Identity Strategy’ – Strategy to identify users. Possible values are USE_DN and USE_USERNAME. The default functionality if this property is missing is USE_DN in order to retain backward compatibility. USE_DN will use the full DN of the user entry if possible. USE_USERNAME will use the username the user logged in with.
‘Authentication Expiration’ – The duration of how long the user authentication is valid for. If the user never logs out, they will be required to log back in following this duration.

OK, enough theory, let’s install a LDAP server using Apache Directory Studio. This project provides an easy way to setup a LDAP server but is also providing a great GUI to manage/administrate existing LDAP servers.I’ll go quick because it’s quite simple to setup and if needed the documentation of the official website is very useful.

Once downloaded and installed, just launch it. On the workbench, we are going to create a new server. Click on the ‘+’ symbol in the “LDAP Servers” tab:

screen-shot-2017-01-24-at-10-04-01-pm

Then, select Apache DS and give it a name:

Screen Shot 2017-01-24 at 10.04.16 PM.png

Create a connection: right click on your server / create a connection. And start your server to access it. You should be able to access the Overview tab of your server. We are going to create a partition/branch for NiFi users:

Screen Shot 2017-01-24 at 10.04.52 PM.png

Click on Advanced Partitions configuration and then Add a new partition. Here I decided to call my partition “dc=nifi,dc=com”:

Screen Shot 2017-01-24 at 10.05.14 PM.png

At this point, you need to restart your server (right click / stop, right click / start).

Now we are going to create an organizational unit for groups and an organizational unit for people. In the ou=groups, we will define two groups, one for normal users and one for administrators. And we are going to create one user in each group, a user “test” in the group “users”, and a user “admin” in the group “admins”. This can be done through the GUI but in this case, I’ll do it by importing the below LDIF file:

dn: ou=people,dc=nifi,dc=com
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people

dn: ou=groups,dc=nifi,dc=com
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups

dn: cn=users,ou=groups,dc=nifi,dc=com
objectClass: groupOfUniqueNames
objectClass: top
cn: users
uniqueMember: cn=test,ou=people,dc=nifi,dc=com

dn: cn=admins,ou=groups,dc=nifi,dc=com
objectClass: groupOfUniqueNames
objectClass: top
cn: admins
uniqueMember: cn=admin,ou=people,dc=nifi,dc=com

dn: cn=test,ou=people,dc=nifi,dc=com
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: test
description: A test user
sn: test
uid: test
mail: test@nifi.com
userpassword: password

dn: cn=admin,ou=people,dc=nifi,dc=com
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: admin
description: A admin user
sn: admin
uid: admin
mail: admin@nifi.com
userpassword: password

To import it, right click on dc=nifi,dc=com, then Import, then LDIF import and select your file.

This will give you the following structure:

Screen Shot 2017-01-24 at 10.27.40 PM.png

Now we want to configure NiFi to connect to our LDAP server. For that you have to note that, by default, the manager of the server (for an Apache DS LDAP server) has “uid=admin,ou=system” as DN and “secret” as password. Then the XML file is configured as below (no LDAPS/TLS in this example):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
    <provider>
        <identifier>ldap-provider</identifier>
        <class>org.apache.nifi.ldap.LdapProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>

        <property name="Manager DN">uid=admin,ou=system</property>
        <property name="Manager Password">secret</property>

        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>

        <property name="Url">ldap://localhost:10389</property>
        <property name="User Search Base">ou=people,dc=nifi,dc=com</property>
        <property name="User Search Filter">uid={0}</property>

        <property name="Identity Strategy">USE_USERNAME</property>
        <property name="Authentication Expiration">12 hours</property>
    </provider>
</loginIdentityProviders>

We need to restart NiFi to take into account the modifications. Note: if NiFi is clustered, configuration files must be the same on all nodes.

Now… if you try to connect as test or admin, you will get the following error:

Unknown user with identity ‘admin’. Contact the system administrator.

This is because you first need to add this user in the list of users through NiFi UI using the initial admin account (see Apache NiFi 1.1.0 – Secured cluster setup). At there is no syncing mechanism to automatically add LDAP users/groups into NiFi.

When connected with your initial admin account (using your individual certificate), go into users to add your users, and then into policies to grant access and rights to the users:

Screen Shot 2017-01-24 at 10.45.35 PM.png

Screen Shot 2017-01-24 at 10.45.46 PM.png

You have now a NiFi instance integrated with a LDAP server and you can connect as different users defined in your LDAP. It gives you the opportunity to add users and play with the policy model implemented in NiFi.

Important note: NiFi has a large and active community, new features regarding LDAP integration could be provided very soon (for example: NIFI-3115).

As always, comments/remarks are welcomed!

25 thoughts on “Integration of NiFi with LDAP

  1. After configured the xml, I restart nifi. Then I get this Exception.
    Caused by: org.apache.nifi.authentication.exception.ProviderCreationException: TLS – Protocol must be specified.

    Can u tell me how can i start with no TLS-Protocol?

    Like

      • Thanks a lot for your reply. It turns out that my LDAP configuration is wrong. Now I can login nifi sucessfully. But there is a new problem. I create a user named ‘admin’ and I can access NiFi web ui after logged in with ldap user ‘admin’.

        It shows “Unable to perform the desired action due to insufficient permissions. Contact the system administrator.” in the page.

        What should I do next?Sorry for asking too much, I am pretty new for nifi. Wish your reply

        Like

      • Basically, setting up LDAP is not enough, it allows you to authenticate against NiFi but, by default, all the users from your LDAP directory have no rights at all. You need to connect to NiFi as the “initial admin user” (using a client certificate – no password) and then to add the user in the NiFi users (NiFi menu / Users) and then to grant the permissions to this user (NiFi menu / Policies). Let me know if it does not make sense to you.

        Like

      • Thanks, I connect to NiFi as the “initial admin user”. Then a add a user named “admin” then a grant the permissions to admin. This is my “authorizations.xml” file

        This is my “users.xml” file

        nifi-user.log like this

        Authentication success for cn=admin,ou=people,dc=nifi,dc=com
        o.a.n.w.a.c.AccessDeniedExceptionMapper cn=admin,ou=people,dc=nifi,dc=com does not have permission to access the requested resource. Returning Forbidden response.

        Like

      • Once your user is added in Users, you should grant the permission “view the user interface” to this user in Policies. It’ll allow you to get on the NiFi canvas with your user (but nothing else, unless you grant the correct other permissions).

        Like

      • I granted the permission “view the user interface” to user “admin” in Policies menu. Then I restart nifi, but the problem still exists. nifi-user.log file:AccessDeniedExceptionMapper cn=admin,ou=people,dc=nifi,dc=com does not have permission to access the requested resource.
        Any suggests? Thanks in advance.

        Like

      • Can you share authorizations.xml file, users.xml file in a gist (gist.github.com) online? Also what did you configure for “Identity Strategy” in your LDAP configuration? Can you try to add a user using the full DN instead of just admin and see if it’s better?

        Like

      • I’m really sincerely grateful for all your help! My problem is solved now. Wish you a have good day!

        Like

  2. After all changing on xml, I got this error below:

    2017-06-12 20:53:19,245 INFO [NiFi Web Server-17] o.a.n.w.a.c.IllegalArgumentExceptionMapper java.lang.IllegalArgumentException: The supplied username and password are not valid.. Returning Bad Request response.

    I already check all the configuration, but still work. Follow all the examples available on the internet, and documentation, but still having error on nifi-user.log.

    Can u help me?

    Like

  3. Hello pvillard31 , i have got any error
    creating bean with name ‘loginIdentityProvider’: FactoryBean threw exception on object creation; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /hxxx/bizxxxxxime/nifi-1.3.0/./conf/login-identity-providers.xml
    [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server…
    o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@caee26e{HTTP/1.1,[http/1.1]}{0.0.0.0:9090}
    org.eclipse.jetty.server.session Stopped scavenging
    Can i know wat reason

    Like

  4. Hi,
    I’m trying to integrate Nifi with Ldap. I made necessary changes as given above. Now nifi UI is working fine but I do not get the login page. Please suggest, where I’m missing the configuration.

    ldap-provider
    org.apache.nifi.ldap.LdapProvider
    LDAPS
    uid=nifiadmin,ou=nifi,dc=abc,dc=com
    nifiadmin-password
    /usr/hdf/current/nifi/conf/keystore.jks
    changeit
    JKS
    /usr/hdf/current/nifi/conf/truststore.jks
    changeit
    JKS
    REQUIRED
    TLS

    FOLLOW
    10 secs
    10 secs
    ldaps://ldap-dallas.abc.com:8443
    cn=nifiadmin,ou=nifi,dc=abc,dc=com
    sAMAccountName={0}
    12 hours


    Also, How to verify that Nifi is successfully integrated with ldap or not?

    Like

    • Best is to try authenticating with a user from LDAP. Note that appropriate authorization need to be set for this user if you want to access the canvas. Authorizations can be set using the initial admin user.

      Like

  5. Hi,

    In addition to above query:
    1. I do not see users tab in Nifi UI main menu.
    2. By default it logins with anonymous user. I can see at the Log tab just beside to main menu.. After selecting login tab and putting credentials into that it gives error as given below:
    “Unable to validate the supplied credentials. Please contact the system administrator.”
    Complete configuration posted in
    https://community.hortonworks.com/questions/139404/how-to-integrate-nifi-with-ldap-by-using-ranger-po.html
    Note: Ranger is integrated with LDAP and able to login ranger UI trough domain users.

    Please suggest if anything missing through.

    Like

  6. Complete setup scenario:
    In a cluster ( HDF 3.0.1 – Ambari, Nifi, zookeeper, Ranger, DB – Mysql ), all componants are running fine. Nifi UI is configured with HTTPS but do not get successful login page in Nifi UI.
    (To configure Nifi UI with HTTPS – converted keystore.jks file into pks12 format and loaded the pks12 file into browser)

    Ranger is integrated with LDAP successfully. Ranger UI is accessible through LDAP users.
    Copied Nifi’s keystore and trustore file from Nifi server to Ranger server to build the trust between them. (copied at /usr/hdf/current/ranger-admin/conf) Then Ranger Policy is created and added LDAP users in it. Also given Read and Write permissions to added LDAP users in Ranger policy.

    Now there is one issue. If I add some LDAP users in the Ranger policy then I cannot access Nifi UI. I got ‘insuffecient permissions and unable to access the page’ kind of errors. Logs shows authentication is success for LDAP users but authorization is failed.

    But If I gave {users} in user’s section of Ranger Policy, then I can login Nifi UI with my LDAP user. Also Nifi UI can be accessible by anonymous user. I dont know from where anonymous user is coming.
    But If I remove {user} from user section then I cannot login with LDAP user as well as anonymous user.

    As per some blogs, I found it could be the related from authorizations.xml and users.xml files. But those files are missing from Nifi servers.
    How to create/generate those files ?

    Nifi Config

    nifi.security.user.login.identity.provider
    ldap-provider

    Template for login-identity-providers.xml

    ldap-provider
    org.apache.nifi.ldap.LdapProvider
    USE_USERNAME
    SIMPLE
    CN=zxc_oi,OU=fox,DC=abc,DC=com
    xxx
    FOLLOW
    10 secs
    10 secs
    ldap://ldap.abc.com:389
    DC=abc,DC=com
    sAMAccountName={0}
    12 hours

    Like

      • Thank you @pvillard31

        I found the solution. Issue is fixed now.

        In my case, one of LDAP username is ‘dvteam’ but in LDAP database there was full description of username as ‘architecture dev team, locations, team details, etc’.

        Error messages I found in nifi-user.log. is ‘architecture dev team’ user was trying to authenticate with nifi nodes. Authentication was successful but authorizations not happening.

        The username which I’ve mentioned in initial admin identity was ‘dvteam’.(cn=dvteam,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Then as per logs, I changed it to (cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com)

        Also there was some mismatch about host names in node identities section. ‘hostname -f’ shows a hostname ip-zz-xx-ec2-internal. So, I have given ‘ip-zz-xx-ec2-internal’ in node identities section but that was not working. Then I have changed the hostnames to ‘nifi1.abc.local’ and mentioned in node identities.

        In ‘Template for login-identity-providers.xml’ I’ve made some changes. Earlier I had set ‘use_username’ in ‘USE_DN’ this section.

        later I’ve changed to use_dn. because as per nifi-user log authentication is happening with LDAP user ‘architecture dev team’.

        So in my case user_username was not working for authentications.

        Every configurations changes I used to remove authorizations.xml and users.xml file from my all nifi nodes.

        Also There was confusion on about ‘OU’ in Node identities section.

        What does it mean OU in node identities section? I don’t know yet.

        Later I’ve mentioned ‘OU=nifi’ and also gave host names as ‘nifi1.abc.local’ , ‘nifi2.abc.local’, etc.

        I have added AD/LDAP user in Initial Admin Identity(cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com)

        After setting above all, I was facing an error about setting nifi.security.identity.mapping.pattern.dn.

        There was a challenge about the pattern definition.

        There was 4 ‘ou’ I have defined in initial admin identities and login-identity-providers.xml.

        So I’ve used below pattern and it worked well.

        ^cn=(.?),ou=(.?),ou=(.?),ou=(.?),ou=(.?),dc=(.?),dc=(.?)$
        Note: I have removed Ranger completely.

        Thanks,

        Suraj

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s