Apache NiFi is easily extensible through the development of custom components. In this Apache NiFi tutorial, you will learn-
What is Apache NiFi?
Why Use Apache NIfi?
History of Apache NiFi
NiFi Architecture
Apache NiFi Features
How to Install Apache NiFi
Nifi Use Cases
Best practices Running Apache NiFi
Disadvantage of Nifi
Why Use Apache NIfi?
Here, are reasons for using Apache Nifi:
Allows you to do data ingestion to pull data into NiFi, from numerous data sources and create flow files It offers real-time control which helps you to manage the movement of data between any source & destination Visualize DataFlow at the enterprise level Provide common tooling and extensions Allows you to take advantage of existing libraries and Java ecosystem functionality Helps organizations to integrate Nifi with their existing infrastructure NiFi is designed to scale-out in clusters which offer guaranteed delivery of data Visualize and Monitor performance, behavior in a flow bulletin which offers insight and inline documentation Helps you to start and stop components separately or at the group level It helps you to listen, fetch, split, aggregate, route, transform and drag & drop Dataflow
History of Apache NiFi
Developed at NSA for over eight years 2014- It was donated to the Apache Software Foundation 2015- NiFi became an official part of the Apache Project Suite Since then every 6-8 weeks, Apache NiFi releases a new update
NiFi Architecture
Apache NiFi has a well-thought-out architecture. Once data is fetched from external sources, it is represented as FlowFile inside Apache NiFi architecture.
NiFi Architecture Here are key components of NiFi architecture
Apache NiFi Features
NiFi supports buffering of all queued data and offers an ability of back pressure as those queues may reach specified limits NiFi allows the setting of one or more prioritization schemes Provides connection processors for many data sources Support any device which runs Java Ideal for limited connectivity places Support for troubleshooting and flow optimization Offers role-based authentication/authorization Allows download, recovery, and replay of individual files Build your processors, controller services, and more Provide content encryption, communication over secure protocols Enables rapid development and effective testing Allows for the development of simple single-function components that can be reused and combined to make more complex flows Allows classloader isolation for easier management of dependencies
How to Install Apache NiFi
Below is a step by step process for Apache NiFi installation Step 1) Go to the link, and click “Continue to Subscribe”
Step 2) On the next page, Click “Accept Terms”
Step 3) You will see this page, Thank you for subscribing to this product! We are processing your request.
Step 4) Refresh the page after 5 minutes. Click on “Continue to Configuration”
Step 5) On the next page, Keep settings default and click “Continue to Launch”
Step 6) On the next page, Click on Launch. You may need to create a key
Step 7) You will see this success message. Congratulations! An instance of this software is successfully deployed on EC2!
Step 8) Note, The instance id and the public DNS of the EC2 instance
Step 9) In the security group, Add all traffic rule to inbound and outbound
Step 10) To access Nifi, simply use the URL In our case it becomes User: admin Password: Instance Id
Step 11) You will see, NiFi home screen
Nifi Use Cases
Below is a list of Apache NiFi use cases:
Best practices Running Apache NiFi
Ideal to separate test/dev/production environments in NiFi You should break your flow into process groups Use a naming convention, use comments and labels Organize your projects into three parts ingestion, test & monitoring Use unique names for variable
Disadvantage of Nifi
Need precise security and compliance controls You need to know the underlying system very well while working with Apache NiFi Must maintain chain of custody for data Transport / Messaging may not prove enough Data access needs exceed available resources to transport Not all data is created equally SSL and topic level authorization may not be sufficient
Summary
Apache NiFi is an open source software for automating and managing the flow of data between systems NiFi is designed to scale-out in clusters which offer guaranteed delivery of data Nifi was developed at NSA for over eight years Once data is fetched from external sources, it is represented as FlowFile inside Apache NiFi architecture. FlowFile, processor, controller, web server, connection, back pressure, repository are important components of NiFi architecture NiFi expression language supports any device which runs Java You can easily install NiFi on AWS NiFi is used in varied industries such as healthcare, insurance, telecom, manufacturing, finance, oil and gas among others As a best practice, organize your projects into three parts ingestion, test & monitoring
