0x3eb

  The last article introduced some common stream processing challenges. In this article, I will try to describe some stream processing designs that can fit as solutions for some of the common stream processing problems. Programming: How the stream processing system/application deals with different aspects of processing is important, and this is instructed by the program made by the developers. There are two different paradigms of programming that can be considered for the stream processing solutions. The two paradigms are known as imperative and declarative programming. Using both of these programming paradigms, we define what should be done. In imperative programming, we have to make the extra effort of describing how an operation should be carried out. Each of these paradigms are described below on how they fit into stream processing. i.) Imperative Programming This approach expects us to develop a custom application that tackles all the challenges discussed in the preceding arti...

Stream processing involves reading from at least one unbounded source. Unbounded sources are sources which do not have any defined beginning or end. Data that keeps arriving is continuously considered in the computation of the result. Whereas the normal batch processing involves reading the fixed-size data completely and only then following the read process with computations. The results are generated once for every operation in a job. In the case of an unbounded source, as the data does not have a determined end, the processing system cannot wait for the entire data to arrive in order to process. To tackle the problem that unbounded source systems pose, stream processing systems provide multiple approaches, which will be much of the focus in this article.   One common processing pattern involves logically group elements into fixed-interval windows based on the time that they were created(in the source system)/received(in the processing system). Generally, for each window, the comp...

Datasets are one of the APIs provided in the Spark’s Structured API. It is known as the foundational type of the structured APIs, as dataframes themselves are a type of dataset. Datasets API is a JVM language feature,hence it works only with Scala and Java. Datasets are also known as typed set API. The  name is due to the fact that datasets are strongly typed, to be more specific they are domain-specific typed objects. They can be operated on in parallel using functional programming constructs or operations that we are familiar with the Dataframe API.   Datasets are represented by the following notation: Dataset[T] Where T is a domain-specific type defined using either Scala Case classes or JavaBean Classes. This type can contain multiple attributes, which acts as columns in the datasets. Each row in the dataset will be an object of class T. One example case class can be found below case class Product(pId: Int, product_name:String, category: String, price: Double) The ca...

  It is possible that there might be situations where there aren't some transformations available in Structured API that are required for a use case that one is working. To deal with those situations, Spark allows us to define user-defined functions(UDFs). Spar allows us to write our own custom transformations using Python, Scala and Java. These UDFs can involve usage of external libraries as well. So with these functionalities, possibilities are endless and the transformation toolkit is extensible.  UDFs in Spark can take one or more columns as input and return one or more columns as output. They can be written in different programming languages and be used in the same programming languages and other supported languages. So an UDF written in Scala can be used in transformations in PySpark. UDFs are functions that operate on data record by record. To use a UDF within Spark, one should follow the following steps: We are supposed to write a function which will take necessary num...