sean cassidy : Write your own Data Structures

in: programming

One of the fundamental skills a software engineer needs in order to be successful today is a mastery of Google-fu. If you do not reference the accumulated knowledge and experience of the Internet on a daily basis your work will suffer. However, I think we, as a community of engineers and thinkers, should take a step back when someone says, "The best programmers are the first to Google."

What this, in essence, says, is that there is little-to-no benefit to learning tricky algorithms and complicated data structures. That instead, you should focus your work on using these tools to solve problems. What is missed is that by knowing how these algorithms and data structures work, on a fundamental level, will improve your ability to solve problems dramatically. Further, you will be less likely to stray into unknown unknowns and the bad side of the Dunning-Kruger effect.

The Dunning-Kruger effect is the phenomenon that the less knowledgable you are, the more you think you know. In effect, you fail to recognize the scope of knowledge. There is so much stuff to learn, that merely by learning more, you recognize how little you know. When I look at Coursera, for example, I often must stop myself from signing up for every other course. There's so much to learn, and so little time to learn it all. One thing is for sure: you'll never be one of the best programmers if you do not take the time to learn.

So, when you choose to download someone's solution to rate limiting, say, you have missed a learning opportunity. This is sometimes of no concern, with deadlines what they often are. But if it becomes the main way you work, you will be a victim of the bad side of the Dunning-Kruger effect: you will begin to not even know certain solutions are available. You would not have missed the obvious application of a Bloom filter nor would you have attempted to breadth-first search a cyclic graph if you had spent more time learning and less time copy pasting.

My advice is to take some time to implement a data structure in your language of choice. Pick one that isn't in your standard library, but might be useful to you. You'll learn several things:

  1. Data structures need to be fast, fast, fast
  2. It's harder than it looks
  3. How to write a proper API with contracts
  4. Correctness is paramount
  5. Writing tests to complete code coverage

And, it's fun. That's why I have a small project where I collect data structures I've written in C. I learned a great deal from implementing them, and I've used them several times since.

One last word: of course you should use all the tools available to you, and you'll never have time to implement every data structure and algorithm yourself. Should you use your own "thread-safe" queue in production instead of Doug Lea's excellent ConcurrentLinkedQueue? No. Try it for fun, and maybe you'll be the next Doug Lea. He certainly didn't copy and paste that code from someone else.

Sean is the CTO and co-founder at DefenseStorm, a cloud cybersecurity startup.

Follow @sean_a_cassidy