On API Design and Simplicity

Originally published at: https://medium.com/@kedars/on-api-design-and-simplicity-305f8dc45fa1

Most of my experience has been with software written in C, so this is primarily for the C language. I never thought this post would be so long, but once I started writing, stuff kept coming up. Its a long read: not how the API documentation should be ;-)

For a software module, its API is the gateway for using the module. It could often be a make-or-break for the module. I feel very strongly about this, so here are some of my experiences with designing APIs.

1. It needs attention

In early part of our career, we don’t care much about the API we expose. The first functions that we write become the API. How about changing this API, someone asks. Meh, it gets the job done, we say, we are too set in our ways using them as they are.
I think making a simple and intuitive API plays a big role in the success of any project. Think Arduino, for example. Every additional line required must be questioned (do you really need to include all of those header files?), every parameter argued.
The API just doesn’t show up. Someone has been obsessing over it. Iteratively polishing and refining it.
I wished I realised this earlier. The API design needs careful attention.

2. Punishing 90% of the users

As developers, we are proud of the ingenious ways in which we implemented an advanced feature, rightly so. And we want to world to know about the ingenuity. After all, what good is an engineer if she is not proud of her deliverables.
We stretch our API to accommodate these advanced features. And soon, this starts to interfere with the simplicity of the API. Turns out, most users don’t even bother about that feature. But by making it a core part of our API experience, we end up punishing these majority of users.
The users of our API now have to use some arcane API, thanks to that advanced, but-not-often-used feature.
What we should be doing instead is to focus on those 90% of the users. Make sure we make their lives easy.

3. Incremental Education

A filesystem allows us to read and write data from a file, all by using only the calls open(), read(), write() and close(). Internally, it is has to manage caching, directory lookups, optimising the block device driver reads/writes etc. But the user doesn’t necessarily have to worry about all of it.
Most APIs expose abstractions. And abstractions end up being leaky. Of course, there will be some users of the API that want to control that internal setting. They would want to control blocking/non-blocking behaviour, lock parts of the file or duplicate file descriptors.
But they don’t have to worry about learning those things, until they have a need for it.
This is what I mean by incremental education. The primary API is simple. And then there are additional APIs for the advanced features. You don’t need to know them, until you need them.
The obvious question that pops-up is, is it always possible (and wise) to abstract out the configurability and tunability that our module provides? One way to do it is to provide reasonable defaults.

4. Reasonable Defaults

We generally write software modules that implements mechanism without locking-in any policies within them, letting the users of the API choose that policy. This is the right approach.
However, it helps to have simpler APIs where default policies are chosen for the user. These policies are the default versions that we think 90% of our users should be ok with. And then there are the advanced APIs to override these policies.
For example, the UNIX filesystem API assumes the reasonable defaults that the read/write operation should be in the blocking-mode and caching should be used.
So the reasonable defaults do the Right Thing and are safe.
Defining what reasonable defaults are, can be quite tricky to begin with. But over a few iterations through our customers, the direction usually becomes quite clear.

5. “Convenience” Layers

As you may have realised in all of this, we want to make it as simple as possible, without having to lose as much flexibility as possible.
The model that, I thought, worked best is where we have multiple layers of convenience APIs.
The topmost layer provides the simplest API to the user as described above.
A user that needs a different behaviour, can peel these layers and use the layers below. She will have to learn a bit more, since the lower layer APIs are more flexible (and hence not as simple).
What needs to be ensured in this case, though, is a clear map of the various software layers and their dependencies on each other. As long as this block diagram is clear and easily available, a user can choose the layer at which they wish to operate for a given functionality.

6. Knowing Your Users

You might have noted that quite a number of methods above, mention ‘iteration’ or ‘as required by majority of our users’. Developing the module in isolation is hard indeed.
While we start off by a certain set of users in our minds to begin with, the user profile changes. It helps to stay in close quarters with the users, understand their current problems, and importantly, close the feedback loop and evolve our API.
This is quite obvious in most open-source projects, but becomes a hurdle in other projects with multiple intermediate layers of support involved.

7. Longevity

And finally, since we are talking about iteration, what about backward compatibility? Software will continue to evolve, impacting the API one way or the other.
Users are typically understanding of API changes as long as we explain the rationale for the API change. And if it makes the life of 90% of the users easy, more power to you.
As folks get into production though, there will be a resistance on the part of the users to change. You gotta bite the bullet and maintain that backward compatibility with the older not-so-simple API.
But the process of evolution and simplification continues, after all, we will have many more users in the future than we’ve had so far, wouldn’t we?

8. Patterns

I thought I was done talking, until it occurred to me that I am missing this point. If you’ve stayed so long, one bonus section for you :-)
Patterns convey information even before having to read the entire details. lock-unlock, open-close, read-write can quickly convey to a developer what might be happening.
Modelling an API around a well established pattern helps users to quickly associate and understand what the API intends to do.
The real danger though is an API that looks like a pattern but isn’t really.

---

Let me know what you think in the comments below…

---

Thanks to Amey Inamdar for his reviewing a draft of this. Thanks to pixabay and freeimages for the images.

The Frog in Hot Water

You identify a niche, you build a product.

You talk to a few early customers, they try it out.

As a startup you have a zillion things to do. Out of the many axes of optimisation, you optimise the few that you want to strategically focus on.

Your customers love it, it solves a big problem for them. They are excited about it.

The good word about your product spreads and you attract more customers. Soon you have a healthy growth rate.

And then it happens. What your customers thought were minor inconveniences have snowballed into a major issue stream with your product.

Everything was going well, so what exactly happened?

As your product evolves, your customer set evolves too. The early adopters were more adaptable, because it fixed a pain point for them. As your product becomes mainstream, the customers that you attract, and their expectations are mainstream. They expect things on the axes that you didn’t intentionally optimise for.

What you choose to do when faced with this solution is up to you and your product’s domain. But it helps to be cognisant of the fact that this may happen, and look out for it before it is too late to leap out of the pot.

How often are you in the ‘flow’?

You are coding away. A minute goes by… You check the watch, and its been two hours!

You don’t know how those two hours went by, but they were very productive. You feel good. Happy. A joy that comes of active creation.

You, my friend, were in the flow. It’s a state that is a reward in itself, a state with your highest of performance.

Given that, can we try to create conditions to get into that state as frequently as possible? What works for you? How often do you find yourself in the flow?

I find myself at my productive best when I can get at least two sessions of complete concentration in a day. These sessions don’t include checking email, or Facebook, or having design discussions or reviewing other’s work. They include working by myself, typically on a programming assignment. And often it happens when the stuff is churning in my mind for quite some time. So once I sit down, all those thoughts just spill over through the keyboard onto emacs.

What is “success” to you?

Different people, different definition.

Same person, different circumstances, different definitions.

What is success to you right now?

Work for a big label company? Create something that is highly valued/appreciated? Make a lot of money? Feel happy about what you are doing? Buy that BMW that stole your heart? Be indispensable at what you do? Be socially appreciated? Being happy?

AND, are you achieving success regularly? If you haven’t yet, is what you are doing today bringing it any closer?

It helps to revisit every once in a while.

If you do, you know what you have to do today, to reach your definition of success.

If you don’t, you keep envying others for what you don’t even want (the social perception of success).

On Startup Books

(Originally published at: https://medium.com/@kedars/on-startup-books-56c1b9ac8e7#.sp4sjixpj  . Copying errors may exist.)

While many are worth mentioning here, the following 4, I believe, are some of the must reads:

The Lean Startup

The way Eric Ries highlights the pain he felt on spending time and energy working on things that their team thought was valuable to their customers, and the solutions he proposes to shorten that path are very easy to relate to. The concept of avoiding wastage by being very close to your customer (starting with a paying customer) rather than being boxed by sitting on your desk is well put forth.

He also makes a case for not only applying the principle initially, before your get to product-market fit, but throughout your engineering cycle as your product matures. It is the development of an experimental attitude rather than a list of bullet points.

    “If you are not embarrassed by the first version of your product, you’ve launched too late.” — Reid Hoffman

Release early, get feedback, improve and iterate.

Word of caution: Make sure you balance the vision v/s customer-feedback.

    “If I had asked people what they wanted, they would have said faster horses.” — Henry Ford (or maybe not)

The Innovator’s Dilemma

While The Lean Startup talks about avoiding wastage by listening to your customers, this books lists how listening to your customers makes companies miss out on the technological shifts that disrupt the landscape. It is a great book that highlights the advantages of being small and new over big and established. And captures quite a few studies about how established incumbents were displaced with these technological shifts. It is a great book for all startup founders as well as employees looking for new ideas with tiny markets.

My few key takeaways:

•     Disruptive technologies are often characterized by poorer performance for the mainstream attributes. But they are attractive to a tiny set of customers. Thus they open new markets with different value proposition.
•     Large companies don’t want small markets. And their mainstream customers aren’t interested in the new value proposition. So they don’t invest in disruptive technologies.
•     With high pace of technological progress, disruptive technologies soon surpass established (sustaining) technologies and are appealing even to mainstream customers.
•     Mainstream customers, now, as if all of a sudden, wish to move to the newer disruptive technology that the large company has been ignoring.

And finally, even if the large company invested in disruptive technology, it might do so with a sustaining technology mindset, thus setting up for failure from the get go.

The Hard Thing about Hard Things

Ben Horowitz writes about their journey through Opsware in this book. This is a must read by any founder or startup employees.

They came quite close to the brink a few times as they were going through this experience. It emphasizes how any startup requires improvisation. Particularly, even when they had Ben and Marc (with Netscape success already behind them) as a core part of the process.

For employees who come from non-startup backgrounds, they find the uncertainty, the experimentation, the improvisation too unnerving. And a pivot is really the end of the world for them. These guys don’t know what they are doing they end up thinking. And that helps nobody. All potential startup employees must read this book. Startups aren’t just about getting from 0 to 100, but it is about venturing into the uncharted, defuzzing the uncertainty is the adventure.

Great Management Reference: This also happens to be a great reference book for all things people management. Right from dealing with your best employee who starts feeling entitled, to team culture and accountability and creativity. Always a great book to have close at hand and refer to, in time of crisis or otherwise.

Venture Deals

This is my final recommendation for all startup founders, particularly first-time founders.

It captures details about founder agreements, equity funding process, raising money, term-sheets, fine prints. It even includes samples/templates for the most standard terms that are present in most of these VC documents.

It is a great read for anyone trying to understand how the internals for startup financing work.

A seed of an idea

A seed of an idea. Like many others that you thought of. But this seems different. Something interesting. Something more.

You keep thinking about it. Look at it from all the angles. It looks promising.

You discuss it. Yes, you had already thought of that angle. And you thought of it the other way too. But wait wait, you didn’t look at it in this fashion. Oh and also, how about that possibility? That is probably a better a way of approaching it. Interesting, very interesting…

Did somebody already try that? Was that a success or failure? They probably didn’t execute it right. There are many ways it could go wrong, but you, you can do right.

You build a tiny prototype. Just that core piece that is oh-so-critical. Looking good. Now that you see it work… its amazing. Didn’t think this would be an added advantage. Or is that the primary benefit? Hmm…

You know that you know a lot more about it now, than when you started. You know there is so much more.

You have a good hold of that idea now. Or probably the idea has a hold on you.

It is evolving in your sleep. It gives you sparks of brilliance at 5 in the morning. You shudder out of sleep in ecstasy. You know, you rock!

It occupies you all the time.

Breakfast, lunch, coffee or dinner.

You live the idea.

You breathe it.

The idea goes with you everywhere.

 

The idea takes you places.

Give it thought. Give it time. Give it action. Do it!

IoT: Who’s gonna maintain it

As an end-user, how do you ensure that the maker continues to provide timely updates?

If you are an Android user, you know how long it takes for the latest Android updates to make their way on your device. As a phone maker, it’s a lot of effort to incorporate, test and roll-out the upgrades. And on top of that deal with the support requests of users for which the upgrade affected and broke parts of their work flow. And this is a state of affairs for devices that cost you, the end user, upwards of 300/400$.

Now consider you are an end-user for the smart home. You have multiple appliances/devices in your house that are all smart. So what happens for these devices like plugs, bulbs, locks and such which are priced at sub-100 or sub-50 $ price point. If user’s don’t quickly get upgrades for their high-end smart phones, why would they for their IoT devices? Granted the software complexity for IoT devices is much lesser as compared to Android phones, but so are the margins that the makers made.

One reason for necessitating the upgrade is of course security. Newer attack vectors/vulnerabilities continue to get invented and fixes for these should be upgraded across all the deployed devices.

Another reason is maturity/enhancements. With smartdevices, it seems people are settling into the expectation that the device continues to improve over a period of time (Tesla upgrades car to park itself), since the device keeps getting Over-The-Air upgrades.

What is in it for a maker to continue to provide these upgrades?

Reputation?

Will the makers be motivated to ship faster upgrades to retain their reputation? But this hasn’t been motivation enough for Android phone makers to release faster upgrades. Even the largest of Android brands update too slow.

The makers have to spend a lot of money/effort in maintaining, testing and getting the upgrade out. With time, this cost will come down, but until then this will be a big exercise.

Ecosystem?

The ecosystem owners (Google in case of Weave, Apple in case of HomeKit) do have a strong influence on the makers participating in their ecosystem. Particularly because most ecosystems have been driven by a strong focus towards the end-user: simplicity and security. This is a great mechanism to ensure the devices are maintained well. So a big tick-mark for products supporting ecosystems. But considering there are going to be hundreds and hundreds of makers building devices for these ecosystems, how much vigil can the ecosystems keep?

Device As A Service?

What if the end-user doesn’t pay the price full of the product up-front? What if only part of it is paid up-front, and then the rest you pay over a period of time. Essentially, you purchase a service rather than a device. The maker is incentivized to fix issues or you could stop payment. It sounds interesting.

But a smart home could have multiple devices from various vendors. Keeping track of all our recurring payments across all these will be a task and a half. If there is a program that covers devices from multiple vendors and offers these devices as a service, that would be a simplified interface for the end users. It looks a bit convoluted, but could work out well for the end-user.

Others?

Any other options that solve this problem effectively?

The Dreamz Experience

Recently I attended the project exhibition at my Alma-mater. I was excited to sit through the presentation of a project group. The oh-so-familiar logo splashed on their opening slide:

Like the flashbacks in Bollywood movies, it all flashed in front of my eyes. Thirteen years! Its been thirteen years since the Dreamz Group has been guiding these capstone/final year engineering projects. Doing something consistently and with high quality for that long is certainly something.

It all began in 2003, when a group of us friends thought we should do something to build better engineers coming out of colleges. The decision was to focus on these final year projects, be good mentors to the students. And then identify challenging problems to solve, work on their solutions, and do so by following best practices from the industry.

The initiative was very successful. Our projects won prizes at prestigious institutes, presented papers at conferences like OLS, but importantly I believe it was an awesome learning experience for everyone involved, the mentor and the mentee.

We never intended for it to run this long, so we certainly must have done something right, probably accidentally, along the way. And I have been thinking what made it work, here are the top few. Most of these seem basic common sense, but hey, that’s what made it.

The “Why”?

We did a good job of internalizing why exactly are we doing this. It provided us a great anchor to base our decisions and take on new initiatives. Every year, any session we did with students, we reiterated publicly why we are doing this. We threw the doors open for anybody looking for help to reach out to us. I think these repeated public declarations, helped everyone internalize it. Probably like a pledge but with more earnestness and meaning.

A case in point is how we viewed other project guiding groups that came up over the years. We always aimed to get along well with all these. In fact, for students that had gone through our screening process, we also shared our true feedback about them and made referrals at other places for guiding projects. There is no rivalry in teaching, its a good deed.

Involvement

After a couple of years of the activity, we thought: why should learning (or involvement) stop when the students finish off their projects? What else could you do? Since then the senior students were deeply involved with our screening process, running many rounds all by themselves. Some interested folks also acted as co-mentors for the next batches of students. Regular sync-ups ensured that the same values and ethos carries forward through the process.

In all of this, I think everyone developed a sense of ownership, and a community around this work. We made many changes in our processes and approaches, and almost every one of them came through by suggestions from the newly involved folks. We automatically kept up with the times, because a younger batch kept guiding us :).

Along the way, the baton of organizing and mentoring these projects has gone from multiple generations of students. My hearty thanks to all those who have been involved.

Mutual Respect

I learned many things through this activity, but the biggest take away, no doubt, was people.

While I was at this project exhibition (that I mention at the top), I met many folks who I had the privilege of mentoring. Folks, across multiple batches, few of the best students in the best educational institute, and who have gone on and made a difference.

I don’t meet most for years, but when we do meet, the mutual affection and respect gushes forth like a reborn stream at the sight of first rains. It is these bonds, these relationships that make it so worthwhile.

---

Times change, technologies change, and this initiative will continue to adapt and finally stop at some point. Till then, here’s to hoping more such fun experiences!

Thanks to pixabay for the title image.