Apache NiFi 1.2.0 has just been released with a lot of very cool new features… and I take this opportunity to start a series of articles around monitoring. This is a recurring subject and I often hear the same questions. This series won’t provide an exhaustive list of the ways you can use to monitor NiFi (with or without HDF) but, at least, it should get you started!

Here is a quick summary of the subjects that will be covered:

• Bootstrap notifier
• Logback configuration
• SiteToSite (S2S) reporting tasks
  • Bulletin alerting
  • Disk monitoring
  • Memory monitoring
  • Back pressure alerting
• Workflow SLA
• Ambari reporting task and Grafana
• Scripted reporting tasks

For this series of article, I will use, as a demo environment, a 4-nodes HDF cluster (running on CentOS 7):

• One management node (for Ambari, Ambari metrics, Grafana, etc)
• Three nodes for NiFi - secured using the NiFi CA (provided with the NiFi toolkit)

I’m using HDF to take advantage of Ambari to ease the deployment but this is not mandatory for what I’m going to discuss in the articles (except for stuff around the Ambari reporting task obviously).

I will not cover how setting up this environment but if this is something you are looking after, feel free to ask questions (here or on the Hortonworks Community Connection) and to have a look into Hortonworks documentation about HDF.

I don’t want to write a single (very) long article and for the sake of clarity there is one article per listed subject. Also, I’ll try to update the articles to stick as best as possible to latest features provided by NiFi over time.

Also, if you feel that some subjects should be added to the list, let me know and I’ll do my best to cover other monitoring-related questions.