Friday, December 01, 2017

On API Design and Simplicity

Most of my experience has been with software written in C, so this is primarily for the C language. I never thought this post would be so long, but once I started writing, stuff kept coming up. Its a long read: not how the API documentation should be ;-)

For a software module, its API is the gateway for using the module. It could often be a make-or-break for the module. I feel very strongly about this, so here are some of my experiences with designing APIs.

1. It needs attention

In early part of our career, we don’t care much about the API we expose. The first functions that we write become the API. How about changing this API, someone asks. Meh, it gets the job done, we say, we are too set in our ways using them as they are.
I think making a simple and intuitive API plays a big role in the success of any project. Think Arduino, for example. Every additional line required must be questioned (do you really need to include all of those header files?), every parameter argued.
The API just doesn’t show up. Someone has been obsessing over it. Iteratively polishing and refining it.
I wished I realised this earlier. The API design needs careful attention.

2. Punishing 90% of the users

As developers, we are proud of the ingenious ways in which we implemented an advanced feature, rightly so. And we want to world to know about the ingenuity. After all, what good is an engineer if she is not proud of her deliverables.
We stretch our API to accommodate these advanced features. And soon, this starts to interfere with the simplicity of the API. Turns out, most users don’t even bother about that feature. But by making it a core part of our API experience, we end up punishing these majority of users.
The users of our API now have to use some arcane API, thanks to that advanced, but-not-often-used feature.
What we should be doing instead is to focus on those 90% of the users. Make sure we make their lives easy.

3. Incremental Education

A filesystem allows us to read and write data from a file, all by using only the calls open(), read(), write() and close(). Internally, it is has to manage caching, directory lookups, optimising the block device driver reads/writes etc. But the user doesn’t necessarily have to worry about all of it.
Most APIs expose abstractions. And abstractions end up being leaky. Of course, there will be some users of the API that want to control that internal setting. They would want to control blocking/non-blocking behaviour, lock parts of the file or duplicate file descriptors.
But they don’t have to worry about learning those things, until they have a need for it.
This is what I mean by incremental education. The primary API is simple. And then there are additional APIs for the advanced features. You don’t need to know them, until you need them.
The obvious question that pops-up is, is it always possible (and wise) to abstract out the configurability and tunability that our module provides? One way to do it is to provide reasonable defaults.

4. Reasonable Defaults

We generally write software modules that implements mechanism without locking-in any policies within them, letting the users of the API choose that policy. This is the right approach.
However, it helps to have simpler APIs where default policies are chosen for the user. These policies are the default versions that we think 90% of our users should be ok with. And then there are the advanced APIs to override these policies.
For example, the UNIX filesystem API assumes the reasonable defaults that the read/write operation should be in the blocking-mode and caching should be used.
So the reasonable defaults do the Right Thing and are safe.
Defining what reasonable defaults are, can be quite tricky to begin with. But over a few iterations through our customers, the direction usually becomes quite clear.

5. “Convenience” Layers

As you may have realised in all of this, we want to make it as simple as possible, without having to lose as much flexibility as possible.
The model that, I thought, worked best is where we have multiple layers of convenience APIs.
The topmost layer provides the simplest API to the user as described above.
A user that needs a different behaviour, can peel these layers and use the layers below. She will have to learn a bit more, since the lower layer APIs are more flexible (and hence not as simple).
What needs to be ensured in this case, though, is a clear map of the various software layers and their dependencies on each other. As long as this block diagram is clear and easily available, a user can choose the layer at which they wish to operate for a given functionality.

6. Knowing Your Users

You might have noted that quite a number of methods above, mention ‘iteration’ or ‘as required by majority of our users’. Developing the module in isolation is hard indeed.
While we start off by a certain set of users in our minds to begin with, the user profile changes. It helps to stay in close quarters with the users, understand their current problems, and importantly, close the feedback loop and evolve our API.
This is quite obvious in most open-source projects, but becomes a hurdle in other projects with multiple intermediate layers of support involved.

7. Longevity

And finally, since we are talking about iteration, what about backward compatibility? Software will continue to evolve, impacting the API one way or the other.
Users are typically understanding of API changes as long as we explain the rationale for the API change. And if it makes the life of 90% of the users easy, more power to you.
As folks get into production though, there will be a resistance on the part of the users to change. You gotta bite the bullet and maintain that backward compatibility with the older not-so-simple API.
But the process of evolution and simplification continues, after all, we will have many more users in the future than we’ve had so far, wouldn’t we?

8. Patterns

I thought I was done talking, until it occurred to me that I am missing this point. If you’ve stayed so long, one bonus section for you :-)
Patterns convey information even before having to read the entire details. lock-unlock, open-close, read-write can quickly convey to a developer what might be happening.
Modelling an API around a well established pattern helps users to quickly associate and understand what the API intends to do.
The real danger though is an API that looks like a pattern but isn’t really.

Let me know what you think in the comments below…

Thanks to Amey Inamdar for his reviewing a draft of this. Thanks to pixabay and freeimages for the images.

Tuesday, September 26, 2017

The Frog in Hot Water

You identify a niche, you build a product.

You talk to a few early customers, they try it out.

As a startup you have a zillion things to do. Out of the many axes of optimisation, you optimise the few that you want to strategically focus on.

Your customers love it, it solves a big problem for them. They are excited about it.

The good word about your product spreads and you attract more customers. Soon you have a healthy growth rate.

And then it happens. What your customers thought were minor inconveniences have snowballed into a major issue stream with your product.

Everything was going well, so what exactly happened?

As your product evolves, your customer set evolves too. The early adopters were more adaptable, because it fixed a pain point for them. As your product becomes mainstream, the customers that you attract, and their expectations are mainstream. They expect things on the axes that you didn’t intentionally optimise for.

What you choose to do when faced with this solution is up to you and your product’s domain. But it helps to be cognisant of the fact that this may happen, and look out for it before it is too late to leap out of the pot.

How often are you in the ‘flow’?

You are coding away. A minute goes by… You check the watch, and its been two hours!

You don’t know how those two hours went by, but they were very productive. You feel good. Happy. A joy that comes of active creation.

You, my friend, were in the flow. It’s a state that is a reward in itself, a state with your highest of performance.

Given that, can we try to create conditions to get into that state as frequently as possible? What works for you? How often do you find yourself in the flow?

I find myself at my productive best when I can get at least two sessions of complete concentration in a day. These sessions don’t include checking email, or Facebook, or having design discussions or reviewing other’s work. They include working by myself, typically on a programming assignment. And often it happens when the stuff is churning in my mind for quite some time. So once I sit down, all those thoughts just spill over through the keyboard onto emacs.

Wednesday, March 22, 2017

What is “success” to you?

Different people, different definition.

Same person, different circumstances, different definitions.

What is success to you right now?

Work for a big label company? Create something that is highly valued/appreciated? Make a lot of money? Feel happy about what you are doing? Buy that BMW that stole your heart? Be indispensable at what you do? Be socially appreciated? Being happy?

AND, are you achieving success regularly? If you haven’t yet, is what you are doing today bringing it any closer?

It helps to revisit every once in a while.

If you do, you know what you have to do today, to reach your definition of success.

If you don’t, you keep envying others for what you don’t even want (the social perception of success).

Tuesday, January 17, 2017

On Startup Books

(Originally published at:  . Copying errors may exist.)

While many are worth mentioning here, the following 4, I believe, are some of the must reads:

The Lean Startup

The way Eric Ries highlights the pain he felt on spending time and energy working on things that their team thought was valuable to their customers, and the solutions he proposes to shorten that path are very easy to relate to. The concept of avoiding wastage by being very close to your customer (starting with a paying customer) rather than being boxed by sitting on your desk is well put forth.

He also makes a case for not only applying the principle initially, before your get to product-market fit, but throughout your engineering cycle as your product matures. It is the development of an experimental attitude rather than a list of bullet points.

    “If you are not embarrassed by the first version of your product, you’ve launched too late.” — Reid Hoffman

Release early, get feedback, improve and iterate.

Word of caution: Make sure you balance the vision v/s customer-feedback.

    “If I had asked people what they wanted, they would have said faster horses.” — Henry Ford (or maybe not)

The Innovator’s Dilemma

While The Lean Startup talks about avoiding wastage by listening to your customers, this books lists how listening to your customers makes companies miss out on the technological shifts that disrupt the landscape. It is a great book that highlights the advantages of being small and new over big and established. And captures quite a few studies about how established incumbents were displaced with these technological shifts. It is a great book for all startup founders as well as employees looking for new ideas with tiny markets.

My few key takeaways:
  •     Disruptive technologies are often characterized by poorer performance for the mainstream attributes. But they are attractive to a tiny set of customers. Thus they open new markets with different value proposition.
  •     Large companies don’t want small markets. And their mainstream customers aren’t interested in the new value proposition. So they don’t invest in disruptive technologies.
  •     With high pace of technological progress, disruptive technologies soon surpass established (sustaining) technologies and are appealing even to mainstream customers.
  •     Mainstream customers, now, as if all of a sudden, wish to move to the newer disruptive technology that the large company has been ignoring.

And finally, even if the large company invested in disruptive technology, it might do so with a sustaining technology mindset, thus setting up for failure from the get go.

The Hard Thing about Hard Things

Ben Horowitz writes about their journey through Opsware in this book. This is a must read by any founder or startup employees.

They came quite close to the brink a few times as they were going through this experience. It emphasizes how any startup requires improvisation. Particularly, even when they had Ben and Marc (with Netscape success already behind them) as a core part of the process.

For employees who come from non-startup backgrounds, they find the uncertainty, the experimentation, the improvisation too unnerving. And a pivot is really the end of the world for them. These guys don’t know what they are doing they end up thinking. And that helps nobody. All potential startup employees must read this book. Startups aren’t just about getting from 0 to 100, but it is about venturing into the uncharted, defuzzing the uncertainty is the adventure.

Great Management Reference: This also happens to be a great reference book for all things people management. Right from dealing with your best employee who starts feeling entitled, to team culture and accountability and creativity. Always a great book to have close at hand and refer to, in time of crisis or otherwise.

Venture Deals

This is my final recommendation for all startup founders, particularly first-time founders.

It captures details about founder agreements, equity funding process, raising money, term-sheets, fine prints. It even includes samples/templates for the most standard terms that are present in most of these VC documents.

It is a great read for anyone trying to understand how the internals for startup financing work.