Bridgera IoT – Data Platform

The engine behind Bridgera IoT is a robust IoT platform using Big Data Technologies that originated in Yahoo, LinkedIn, Twitter, eBay, and Google. Our IoT platform brokers the connection between IoT applications and deployed devices (the “Things” in IoT). It’s primary functions are to receive (ingest) and send data, process data, and store data.

I used the word “robust”, not as marketing jargon but as a valid adjective given our IoT platform’s ability to react (process) in real time to extremely high data volumes of any type. This capability is made possible by Big Data technology (and very talented software developers).  

Recently, Joydeep Misra, the lead architect of Bridgera IoT, led a session on data ingestion and processing in an Industrial IoT event sponsored by RIoT in Charlotte, NC. During that session, Joydeep shared how big data technology such as Apache NiFi, Apache Kafka, and Apache Storm was created to read and process data of any schema, while the data are in motion. Which means the data does not have to be written to a database prior to processing, if at all. Equally as important, processing is distributed across computing resources that are instantly available on request, enabling a real time response at any load with built-in redundancy in the event a hardware failure

The big data approach replaces the traditional processing model of a relational database that reads data into a predefined schema (problem #1) then loads data into memory (problem #2) to be accessed by a software program for processing (problem #3).  If you want the technical nitty gritty, read this:

https://www.pearsonitcertification.com/articles/article.aspx?p=2427073&seqNum=2

Problems with Relational Databases in IoT

Problem #1. Reading data from a database prior to processing adds latency and a potential point of failure. Adhering to a  predefined data schema in IoT results in a lack of flexibility when it comes to accepting changes in data structure from deployed devices (bring on the DBA!) and a second point of failure if the structure changes with a device firmware upgrade (there is no human oversight in IoT communications).  

Problem #2. A proper amount of memory must be allocated to each process which may not be cost effective in high volume applications. If the memory limit is reached, the process fails (Out of memory error!).  

Problem #3. The software program containing the processing logic must read from the database, adding latency and highly dependent on database speed. Traditional methods are generally not capable of multi-threaded processing (i.e., handling processing requests simultaneously) or automatic fail-over without significant expense.  

Of course, a traditional software model may be sufficient for IoT solutions with low data volumes, a predefined static data schema, and with up time and response flexibility.  

Data Processing for Beginners

You are at Disneyland. You are data. You request to ride on an attraction (i.e., to be processed). Theoretically, processing should take no longer than the time to walk up to the ride entry point plus the duration of the ride.

In IoT, data processing should take no longer than the time for data to travel from a device to the data center and be processed by a server.  

Thousands crowd on Main Street USA at Disneyland (File photo by: Mark Rightmire, Orange County Register & Southern California Newsgroup)

If you have been to Disney, you know that riders out number the capacity of the rides, resulting in significant queues. To change this, they could speed up the rides, add more rides, or invite less people.

The same holds true for an IoT solution. If the processing capability can not keep up with the device data, you could find a faster processor, add more processors, or connect less Things. Moore’s law tells us the limit to faster processors. Connecting less things may not be compatible with your use case. Therefore, you need a technology that allows you to add more processors (i.e., big data).     

Add More Processors

If there is not a physical limitation to eliminating a queue, there often is a financial limitation. Disney will support just enough ride capacity to maximize ticket sales without disappointing their customers with unacceptable wait times. 

While Disney can clearly get away with queues, many IoT solutions find any wait or processing delay unacceptable. Therefore, as IoT solutions scale, they must be capable of adding more processing in an affordable manner.

Big Data Technology    

Bridgera IoT is IoT software built on Big Data technology. Big Data technology was designed to eliminate unwanted queues in a cost effective manner.

Imagine Disneyland as it exists today with 60 attractions and all 60 attractions have queues. Imagine if Disney could use some of its magic to create a “Disneyland Duplicate”. An alternative park to provide additional “processing” capacity when needed.

Disneyland Duplicate

If Space Mountain were full, you simply transport to the Disneyland Duplicate and ride the exact duplicate of Space Mountain there. Conversely, if DisneyLand were underutilized, Disneyland Duplicate shuts down as if it never existed. The park would operate at optimal “processing” capacity for any volume of people.    

Back to real life. Big Data technology provides Bridgera IoT with a similar capability to Disneyland Duplicate. The technology is designed to spawn processing capacity as needed and where needed. This capacity is then released immediately after use, ensuring a quality of service at any volume of data while eliminating the cost of idle capacity.

As an added benefit, this ability to spawn processing capacity results in stand-by capacity in the event that a server fails. Imagine the quality of service at Disneyland if they could immediately spawn a duplicate ride if and when a ride fails!  

Big Data Storage

We cannot ignore that once the ride is complete, we will want to store the memories. Imagine if Disney wanted to offer a memory storage service but everyone’s memory is different. How would they possibly structure their memory database?  

Bridgera IoT solves this problem by deploying a noSQL database, in our case MongoDB. MongoDB is a document-oriented database rather than table based as found in a relational database. This type of database allows us to store data of any type without first creating a tabular schema.  So if the memories are 1 field (ie, a smile), 10 fields plus a video, a random document, or sight and sound, MongoDB will store and recall them all. 

Bridgera IoT for Big and Small Data

Bridgera IoT provides a quality service at an affordable price by leveraging big data technology to eliminate data processing queues and eliminate foresight needed to develop a relational database schema. Big IoT applications generating Big Data never have to worry about hitting a capacity limit. Mission critical applications can have confidence that an alert will be processed in real time and not suffer the effect of a queue.

Small IoT applications generating small data benefit from Bridgera IoT Software-as-a-Service (SaaS). SaaS provides the highest quality service at the most affordable price. As a SaaS offering, Bridgera IoT treats data ingestion and processing like a utility distributed across many solutions/clients (a lot of small data eventually equals big data). When you need processing capacity, the system provides it. When you don’t need it, the processor shuts down assuring client and data separation between solutions.  

Fill out our contact form if you have questions about Bridgera IoT and or how we apply Big Data Technologies to provide a better service while saving you money and time.

About the Author: Ron Pascuzzi leads sales and marketing at Bridgera, LLC in Raleigh, NC. Ron is an evangelist for the Internet of Things and believes that IoT initiatives should not be compromised due to a lack of software development skills.  Contact him to learn more about Bridgera IoT, not just another IoT Platform but Custom Software-as-a-Service for the Internet of Things.