Do I need to learn functional Programming?

The main problem with writing in verse is, if your fourth line doesn’t come out right, you’ve got to throw four lines away and figure out a whole new way to attack the problem. So the mortality rate is terrific.

Dr. Seuss

x = x + 1

When I started programming in my early graduate years, I found above statement quite puzzling. In normal algebraic context, above line didn’t make sense. But I was told to forget algebra and get used to imperative programming. It’s a statement, understand it and get used to it. During all these years, we got exposed to various scenarios and problems, and we learnt to solve all of them using the tools and techniques of imperative programming. I worked on number of programming languages but all of them were philosophically similar with differences in syntax. I prided myself in being able to pick any language and deliver on short notice.

Enter Scala! I expected to be able to pick the syntax and move on. But to my surprise, soon I realized that it’s new game. Scala comes from a different programming paradigm – functional programming. In the world of functional programming above construct was no more good. It’s whole new way of thinking (Though Scala allows above statement to work because it’s a multi paradigm language, we will keep our focus on functional programming aspect only where it is not a valid construct). Mutation is evil!

Functional programming has its roots in lambda calculus. It has strong mathematical background and  has been more popular in academia than industry. Going functional way will need significant re-tooling of our minds, it has different way of solving problems than imperative. We have to learn new ways of designing and composing programs. It’s going to be decent effort by any standard. Even early adapters of Scala have expressed that there is steep learning curve required for teams to be able to effective.

 

Is functional programming better than imperative programming?

This is also one of those epic battles where no winning side can be declared – imperative programming vs. functional programming (or at least I can’t be that judge!). Both of the sides have equally passionate foot soldiers rooting for it.

I will prefer talking from my personal experience.

A whole new perspective

We tackle variety of scenarios in our day to day programming assignments. They never come with neat labels on them which say it has to be solved with a specific paradigm. All it needs is an elegant solution that is effective and in time. One size doesn’t fit all and it will be good to have all problem solving tools in our toolkit. We will be better by not being held hostage to a particular way of thinking rather open to all. After all our allegiance is with business, not with any specific programming paradigm. Learning functional programming will broaden our perspective.

That is the precise reason why Scala is a multi paradigm language and has support for imperative, functional and object oriented programming. Similar path is being taken by other leading programming languages Java 8, C#, Python. They all are having support for functional programming constructs in varying degrees.

Brevity is the soul of wit

But we are not trying to be funny here! After all programming is a serious business. And we all have heard countless tales of amazing sense of humour of our fellow programmers!

But you have to trust me it is amazing to read a code free of tons of boilerplate code, un-necessary repetitions and obvious type inferences. You can skim the code faster and get hang of purpose of the code way more smoothly and effortlessly in functional than in imperative. It’s amazing to express out computation succinctly and be able to glide though them quickly when needed. By the way it is less error prone as well.

Need for speed

In past few years we have seen a clear trend in hardware that multi core / CPU is the way forward. We are talking about horizontal scaling – distributed programming. Data is exploding and only way to cope it though distributed programming. I find immutability and no side effect philosophy of functional programming specifically effective here. Erlang has been a shining example in this area. Few of recent examples include Spark, Kafka and Flink.

Test all the way

Functional code with its focus on brevity, immutability and referential transparency makes unit and integration testing very easy. It has huge impact on reliability of code and productivity improvement.

 

It’s coming!

It is too early to predict future in the battle of supremacy of paradigms and languages. But one thing is clear to me no matter which camp you belong to (Java/C#/Python etc.), you are going to deal with tenets of functional programming!

So wake up, smile and smell the coffee. Why so serious!

 

How much Scala should a Spark developer know?

Number of times, I come across the point of view that we are Spark developers not Scala developers. We are not required to go deep in Scala. We should know as much as is needed by Spark. I smile and get tempted to ask if there is a formula in their knowledge that determines how much of Scala is needed by Spark!

On a different note, I have a different point of view here. If we want to express our processing needs efficiently and effectively to Spark we better be good in Scala. If we have only rudimentary language skills, our expression will be overly verbose and convoluted. We will try to devise tricks and techniques to achieve the end results with limited Scala vocabulary and grammar we possess. This is error prone, less productive and complicated. Also capabilities and features needed to facilitate our expression might be built into language and might be more stable and efficient than our custom logic that we plan to write.

For example we might use if else construct to achieve decent amount of match feature, but using match will be more elegant and effective. Another example can be case classes, we can achieve similar results with regular classes, but the ease and brevity case classes bring to table is unbeatable.

How testable, extendable and reusable one’s data processing logic is, depends good deal on how well he uses the language features. It’s like having a genie but not knowing enough of his language to communicate your long standing very specific desire.
At the end of the day the key offer from Spark is a collection of items, resilient & distributed. The items might be text document, key value pair or data row. Rest of system is just to keep this collection up and running efficiently and effectively.

I agree Scala has a stiff learning curve and number of adapters have also talked on similar lines. Nonetheless it pays to learn it!