Big data thinking: building for scale and performance

I always been intrigued by big data and the platforms that do the heavy lifting to power the different facets of our lives; social networks, content portals , mobile applications and other services that we use day to day. This interest was spurred by the fact that I was yet to see a truly innovative service emerge from Kenya or Africa at large, that tore through large sets of data and the attendant consumer interactions .

There have emerged those services that in my opinion are starting to get to that big data league that has us thinking more about the hardware and software interaction and what it takes to get services to scale. In my books, Mxit – the South African born mobile social network that currently boasts around 50 million registered users; adding on average 40,000 new users daily and pushing 750 million messages daily and Mpesa that has over 15 million registered users and doing 250 transactions per second give a good starting point for what I call big data thinking.When building services that are targeted for mass consumption whether from an local or international perspective as the two examples represent, a lot of thought needs to be put into the architecture. From experience, this is unfortunately not taught in our various institutions of higher learning and I am not entirely sure that it can. In my experience, this knowledge is built from years of experimentation with success and failure in equal measure and a continuous learning curve, with tweaks to one’s setup as part of a daily endeavor.

There are many questions that you would need to answer as you start visualizing your platform. These can range from; what database and schema to use, language and framework , system to leverage – dedicated or co-located, stack to deploy and tools to address various issues depending on what service you will be running.

A quick way to get up to speed would be to dissect the architecture of successful services that have made their methods and techniques available and adapt these for your deployment or use the learning’s to craft your own unique architecture.

Seamless service delivery across any platform build for mass engagement is dependent on its architecture. And it is imperative that everyone from government, consumer brands, corporates to developers and technology firms targeting mass market solutions; fully embrace the process of “big data thinking” to ensure that any services rolled out are ready for scale and performance and do not become victims of their own success

An Africa based entrepreneur in the pursuit of opportunities without regard to resources currently controlled striving to build services that have real-world value for my beloved continent and beyond while having fun along the way.

Share this: