Sizing Your Apache NiFi Cluster For Production Workloads

Cloudera Flow Management provides an enterprise edition of support Apache NiFi managed by Cloudera Manager. The official documentation provides a great guide for sizing your cluster.

https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-nifi-sizing.html

If the use case fits, NiFi Stateless Engine may fit and perform better utilizing no disk.

Check out that heap usage and utilization, you may need to increase. 24-32 Gigabytes of RAM is a nice sweet spot for most instances.

Check out how your nodes, threads and queues are doing. If queue is not processing fast or thread count is high, you may need more cores, RAM or nodes.

When you are managing your cluster in Cloudera Manager, make sure you increase the default JVM memory for Apache NiFi. 512MB is not going to cut it for anything but single user development.

Do this correctly and process a billion events!!! https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/. - Notice the hardware and performance sections of that article

General tips:

Make sure you use SSD for Provenance and other repositories. Faster disk, happier user. https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-disk-configuration.html

Monitor your flows to see how much resources you need: https://www.datainmotion.dev/2020/07/report-on-this-apache-nifi-1114-monitor.html.

Three or more nodes plus external Zookeepers. https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-cluster-layout.html

Use Records, if it's semistructured GrokReader can help. https://www.nifi.rocks/record-path-cheat-sheet/ If it's CSV, JSON, XML, Parquet, Logs then use Readers and writers. They are much faster, easier and cleaner.

Good clean code is better than spaghetti. https://docs.cloudera.com/cfm/2.0.1/nifi-tuning/topics/cfm-tuning-your-data-flow.html. Saw no to bad coding: https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html.

Recommendations: https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-recommendations.html

Minimize use of CPU or Memory intensive processors (or make a not of them during sizing): https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-resource-intensive-processors.html

There are a few decisions to make on repositories, talk to your Cloudera friends. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/configuration-best-practices.html