Category: Devel

Open Source and Profit

I have written extensively about free, open source software as a way of life, and now reading back my own articles of the past 7 years, I realize that I was wrong on some of the ideas, or in the state of the open source culture within business and around companies.

I’ll make a bold statement to start, trying to get you interested in reading past the introduction, and I hope to give you enough arguments to prove I’m right. Feel free to disagree on the comments section.

The future of business and profit, in years to come, can only come if surrounded by free thoughts.

By free thoughts I mean free/open source software, open hardware, open standards, free knowledge (both free as in beer and as in speech), etc.

Past Ideas

I began my quest to understand the open source business model back in 2006, when I wrote that open source was not just software, but also speech. Having open source (free) software is not enough when the reasons why the software is free are not clear. The reason why this is so is that the synergy, that is greater than the sum of the individual parts, can only be achieved if people have the rights (and incentives) to reach out on every possible level, not just the source, or the hardware. I make that clear later on, in 2009, when I expose the problems of writing closed source software: there is no ecosystem in which to rely, so progress is limited and the end result is always less efficient, since the costs to make it as efficient are too great and would drive the prices of the software too high up to be profitable.

In 2008 I saw both sides of the story, pro and against Richard Stallman, on the views of the legitimacy of propriety control, being it via copyright licenses or proprietary software. I may have come a long way, but I was never against his idea of the perfect society, Richard Stallman’s utopia, or as some friends put it: The Star Trek Universe. The main difference between me and Stallman is that he believes we should fight to the last man to protect ourselves from the evil corporations towards software abuse, while I still believe that it’s impossible for them to sustain this empire for too long. His utopia will come, whether they like it or not.

Finally, in 2011 I wrote about how copying (and even stealing) is the only business model that makes sense (Microsoft, Apple, Oracle etc are all thieves, in that sense) and the number of patent disputes and copyright infringement should serve to prove me right. Last year I think I had finally hit the epiphany, when I discussed all these ideas with a friend and came to the conclusion that I don’t want to live in a world where it’s not possible to copy, share, derive or distribute freely. Without the freedom to share, our hands will be tied to defend against oppression, and it might just be a coincidence, but in the last decade we’ve seen the biggest growth of both disproportionate propriety protection and disproportional governmental oppression that the free world has ever seen.

Can it be different?

Stallman’s argument is that we should fiercely protect ourselves against oppression, and I agree, but after being around business and free software for nearly 20 years, I so far failed to see a business model in which starting everything from scratch, in a secret lab, and releasing the product ready for consumption makes any sense. My view is that society does partake in an evolutionary process that is ubiquitous and compulsory, in which it strives to reduce the cost of the whole process, towards stability (even if local), as much as any other biological, chemical or physical system we know.

So, to prove my argument that an open society is not just desirable, but the only final solution, all I need to do is to show that this is the least energy state of the social system. Open source software, open hardware and all systems where sharing is at the core should be, then, the least costly business models, so to force virtually all companies in the world to follow suit, and create the Stallman’s utopia as a result of the natural stability, not a forced state.

This is crucial, because every forced state is non-natural by definition, and every non-natural state has to be maintained by using resources that could be used otherwise, to enhance the quality of the lives of the individuals of the system (being them human or not, let’s not block our point of view this early). To achieve balance on a social system we have to let things go awry for a while, so that the arguments against such a state are perfectly clear to everyone involved, and there remains no argument that the current state is non-optimal. If there isn’t discomfort, there isn’t the need for change. Without death, there is no life.

Profit

Of all the bad ideas us humans had on how to build a social system, capitalism is probably one of the worst, but it’s also one of the most stable, and that’s because it’s the closest to the jungle rule, survival of the fittest and all that. Regulations and governments never came to actually protect the people, but as to protect capitalism from itself, and continue increasing the profit of the profitable. Socialism and anarchy rely too much on forced states, in which individuals have to be devoid of selfishness, a state that doesn’t exist on the current form of human beings. So, while they’re the product of amazing analysis of the social structure, they still need heavy genetic changes in the constituents of the system to work properly, on a stable, least-energy state.

Having less angry people on the streets is more profitable for the government (less costs with security, more international trust in the local currency, more investments, etc), so panis et circenses will always be more profitable than any real change. However, with more educated societies, result from the increase in profits of the middle class, more real changes will have to be made by governments, even if wrapped in complete populist crap. One step at a time, the population will get more educated, and you’ll end up with more substance and less wrapping.

So, in the end, it’s all about profit. If not using open source/hardware means things will cost more, the tendency will be to use it. And the more everyone uses it, the less valuable will be the products that are not using it, because the ecosystem in which applications and devices are immersed in, becomes the biggest selling point of any product. Would you buy a Blackberry Application, or an Android Application? Today, the answer is close to 80% on the latter, and that’s only because they don’t use the former at all.

It’s not just more expensive to build Blackberry applications, because the system is less open, the tools less advanced, but also the profit margins are smaller, and the return on investment will never justify. This is why Nokia died with their own App store, Symbian was not free, and there was a better, free and open ecosystem already in place. The battle had already been lost, even before it started.

But none of that was really due to moral standards, or Stallman’s bickering. It was only about profit. Microsoft dominated the desktop for a few years, long enough to make a stand and still be dominant after 15 years of irrelevance, but that was only because there was nothing better when they started, not by a long distance. However, when they tried to flood the server market, Linux was not only already relevant, but it was better, cheaper and freer. The LAMP stack was already good enough, and the ecosystem was so open, that it was impossible for anyone with a closed development cycle to even begin to compete on the same level.

Linux became so powerful that, when Apple re-defined the concept of smartphones with the iPhone (beating Nokia’s earlier attempts by light-years of quality), the Android system was created, evolved and dominated in less than a decade. The power to share made possible for Google, a non-device, non-mobile company, to completely outperform a hardware manufacturer in a matter of years. If Google had invented a new OS, not based on anything existent, or if they had closed the source, like Apple did with FreeBSD, they wouldn’t be able to compete, and Apple would still be dominant.

Do we need profit?

So, the question is: is this really necessary? Do we really depend on Google (specifically) to free us from the hands of tyrant companies? Not really. If it wasn’t Google, it’d be someone else. Apple, for a long time, was the odd guy in the room, and they have created an immense value for society: they gave us something to look for, they have educated the world on what we should strive for mobile devices. But once that’s done, the shareable ecosystem learns, evolves and dominate. That’s not because Google is less evil than Apple, but because Android is more profitable than iOS.

Profit here is not just the return on investment that you plan on having on a specific number of years, but adding to that, the potential that the evolving ecosystem will allow people to do when you’ve long lost the control over it. Shareable systems, including open hardware and software, allow people far down in the planing, manufacturing and distributing process to still have profit, regardless of what were your original intentions. One such case is Maddog’s Project Cauã.

By using inexpensive RaspberryPis, by fostering local development and production and by enabling the local community to use all that as a way of living, Maddog’s project is using the power of the open source initiative by completely unrelated people, to empower the people of a country that much needs empowering. That new class of people, from this and other projects, is what is educating the population of the world, and what is allowing the people to fight for their rights, and is the reason why so many civil uprisings are happening in Brazil, Turkey, Egypt.

Instability

All that creates instability, social unrest, whistle-blowing gone wrong (Assange, Snowden), and this is a good thing. We need more of it.

It’s only when people feel uncomfortable with how the governments treat them that they’ll get up their chairs and demand for a change. It’s only when people are educated that they realise that oppression is happening (since there is a force driving us away from the least-energy state, towards enriching the rich), and it’s only when these states are reached that real changes happen.

The more educated society is, the quicker people will rise to arms against oppression, and the closer we’ll be to Stallman’s utopia. So, whether governments and the billionaire minority likes or not, society will go towards stability, and that stability will migrate to local minima. People will rest, and oppression will grow in an oscillatory manner until unrest happens again, and will throw us into yet another minimum state.

Since we don’t want to stay in a local minima, we want to find the best solution not just a solution, having it close to perfect in the first attempt is not optimal, but whether we get it close in the first time or not, the oscillatory nature of social unrest will not change, and nature will always find a way to get us closer to the global minimum.

Conclusion

Is it possible to stay in this unstable state for too long? I don’t think so. But it’s not going to be a quick transition, nor is it going to be easy, nor we’ll get it on the first attempt.

But more importantly, reaching stability is not a matter of forcing us to move towards a better society, it’s a matter of how dynamic systems behave when there are clear energetic state functions. In physical and chemical systems, this is just energy, in biological systems this is the propagation ability, and in social systems, this is profit. As sad as it sounds…

The doctor and the programmer

About 15 years ago, when I was working on a dodgy Brazilian firm, I had a conversation with an older programmer that I never forgot. He said something along the lines of:

Medicine is way easier than computer science. Doctors are still using the same books, written decades ago, while we have to buy only the latest ones. Our reality gets rewritten every five years or so, and by the time you leave university, you’re already outdated.

There is a lot of merit in this argument. Even though the common cold’s strain changes every week, its symptoms are exactly the same. Cancer, HIV, malaria, Lupus and other big diseases are treated more or less the same way as they were when first treated, and GPs still give you aspirin/paracetamol/ibuprofen for any kind of pain.

Human anatomy, physiology and psychology doesn’t change at all. Broken legs, runny noses and swollen tonsils are the same on every person and they require the same treatment for everyone.

While doctors kill one patient when they do make a mistake, developers can kill hundreds if they happen to introduce a bug on the Airbus fault-tolerant fail-over system.

But recently, I had to re-think this through, and I have to say I’m not 100% in agreement any more.

When programmers change jobs, they get a few weeks to get used to the new system. They might get months to actually be productive as their peers and will mature within a few years working with the same piece of software. Programmers can run tests, regression tests and usability tests, unit tests, etc, which is something a bit complicated with human beings.

When a doctor gets a new patient, it’s like getting a new programming job. It’s the same language, but the system is completely different. It might take you weeks to start getting the prognosis right, and in a few months you’ll be able to get it right before your patient even tells you the symptoms.

The similarities are remarkable

Consultant programmers get new systems to work on every week. Like ER doctors. They do what they can, with the time they have and the solution is most of the time acceptable. A more stable doctor or programmer might look at the job and cry, but the job was done, the fire was put off and the “patient” is out of the door.

Family doctors, that were there when you were born, know you better than yourself. They know when your symptoms are only psychological and what cause that and when it’s going to go away. They rarely touch the “system”, but normally fix an unrelated bug and you’re good as new.

But not everybody is lucky enough to have such doctors. They are expensive, and there aren’t enough good doctors in the health system of any country to account for every family. Even if the doctor share a hundred families, it’s still very far from enough.

This is the reason that systems fail, and get half-fixed, and why most GPs will send you home with a paracetamol unless you’re dying in front of them.

If doctors and programmers had such a different world, the emergent behaviour wouldn’t be that similar, I believe.

Why no MMORPG is good enough?

Massively multiplayer online role-playing game (MMORPG) are not new. The first I remember playing is the Legend Of the Red Dragon (LORD), but before that, of course, I’ve played other (real-life) multiplayer RPG games as well, and they were never quite the same thing.

Of course, at that time the graphic cards couldn’t quite compete with our imagination (not to mention connection speeds), but a lot has improved in both fronts, and lots of very good looking games have arrived, but still, there’s something missing. For years I couldn’t put my finger on it, but recently I think I nailed the issue: user driven content.

The interface

Most of the MMORGP are war games. World of Warcraft, LOTR online, Vendetta, Star Trek Online, Regnum and so many others rely on war to be fun. Of course, all of them have the side issues, some trade or silly missions, but the real fun is going to the battlefield.

If you look from the technical side of things, this is not surprising at all. Aside from good graphics, one of the hardest things to do in a game is a good interface. Very few games are perfect like Tetris. I gave Tetris to both my sons when they were about 2 years old and after about a minute they were already playing. There is no need to instructions, tutorials or any training and still today I find it quite fun. This is why it’s probably the most successful game in history.

But it’s all about the interface. It’s easy to create a seamless interface for Tetris. Try to create an interface for a strategy game that doesn’t require some hours of training, or an interface for first-person games that actually allows you to know where you are, or an interface for adventure games that doesn’t make you click in half-dozen places to get anything. Many have tried, all have failed.

At the dawn of internet games, strategy and quake were dominant. That’s because the internet wasn’t so fast and both were quite good in saving bandwidth. Quake had a special fix to avoid sending one packet for every bullet and only one packet when you press the trigger and another when you release it, the rest was up to the client.

But in war games, the interface is pretty much established. World of Warcraft didn’t invent anything, they just mixed Warcraft with Lara Croft (rubbish interface, by the way). Space ship games got the interface from Wing Commander (Vendetta got it from W.C. Privateer), Lord of the Rings and Regnum mixed Second Life with old-style RPG (even with the same deficiencies) and Star Trek Online copied from everyone above.

Now, the interface for a good strategy or adventure game is somewhat complicated. For a first-person 3D RPG, even worse. It doesn’t have to be mind controlled to be easy, nor you have to use 3D glasses or any immersion technology to be fun. Simplifying the game is one way, but then it’s even harder to make it more interesting.

It’s the user, stupid!

I can see only one way to add value to a game that is simple but still fun: user driven content.

You can enrich the world in which you’re immersed into. For instance, Zynga is quickly gathering an impressive amount of users by adding a lot of content. I don’t play those games, but people around me do and I can see why it’s so addictive. There are so many things to do and the frequency of updates is so impressive that it’s almost impossible not to be driven to it.

You might think that the games are too simple, and the graphics are average, but the interface is extremely simple, the challenges are simple, yet still challenging, and the number of paths you can choose for your character are enormous. In this case, the user experience is driven by his own choices. The content is so rich that each and every place is completely different from every other, solely driven by user choices.

Not many game companies (certainly not the indie ones) have time or money to do that. So, why are indie games generally more interesting than commercial ones? They go back to square one, simplify the game and optimise the interface. EA would never release something like Angry Birds or World of Goo, and yet those are the two best games I played in a long time. But, world of Goo is over and Angry Birds require constant attention (seasonal versions) to keep selling or making money (from ads).

They are missing the user content. It might not be their style, nor objective, but that’s a difference between Deep Purple and a one-hit-band.

Back to MMORGP

So, from all MMORPGs, there are many good looking, some with good challenges and a lot of slaughtering fun, but I tire quite quickly from playing them. The last I played was Vendetta. Quite good graphically, it has some reasonably accurate physics simulation (what drove me to it) but not accurate enough to keep me playing. The content tires too quickly to be fun after a few hours and even though I had 8 hours of free play, I spent less than two and dropped it.

This was always a pain, since Final Fantasy and the like, building up the character, hitting slugs for XP, fight-heal-run until you level up. Though Final Fantasy was better, as it normally would throw you on level 10 or so, so you didn’t need too much of levelling up. But why? Who likes beating 253 slugs to get 1000 experience points, going to level 2 and being able to equip a copper sword that doesn’t even cut a snail’s shell?

One of the best MMORGP experiences I had recently was Regnum. This is a free game made in Argentina and has a lot of content, good interface and a healthy community. They do the normal quest levelling up technique and it works quite well until level 15 or so. After that, it’s hitting bears, golems and phantoms for half a year until you can go outside and beat other users.

I got outside prematurely (couldn’t bother to wait) and the experience was also not great. Huge lag on big crowds, people disappearing in mid-air and appearing a few meters away, etc. But the most annoying of all was the content. It was always the same fort that we had to protect, always the same keep we had to attack, always the same talk on how our race is superior to your race, etc.

I haven’t seen Lord of the Rings (which sucks on Linux) or Star Trek Online (which doesn’t even run), but I bet they can get a bit further. They’re there to compete with World of Warcraft, not Regnum, but the fate will be the same: boring.

So, after quite a big rant, how would I do it?

User content

First, a memory refresh: all free first-person shooter I know of are a re-make of Quake. They use the same engine, the same world builders, the same techniques. On Debian repositories you’ll find at least a dozen, all of them running on some version of Quake. Nexuiz, Tremulous, Open Arena, Urban Terror, etc.

Not only the Quake engine is open source, but it was built to be extensible and that, even before the source was opened by ID. I made some levels for Doom, there were good editors at the time (1994?), probably there are full development suites today.

The user has the power to extend, create, evolve and transform your game in ways that you never thought possible. To think that only the few people you hire are good enough to create game content is to be, to say the least, naive.

Now, all those games are segmented. Nexuiz levels don’t connect to Tremulous levels. That’s because the mods (as they’re called) are independent. To be able to play all those different games you need to download a whole lot of data (objects, music, game logic, physics settings, etc) and each game has it radically different. Sarge with a rocket launcher would be invincible in most of other quake variants.

But that is, in my opinion, the missing link between short spurs of fun and long lasting enjoyment. I want to be able t build my world (like Zynga), but in a world with free movement (like Quake) with quests (like MMORPGs) made by the users themselves (like no FP-game I know) in a connected world. It cannot penalise those that connect seldom, or those that connect through text terminals, Android phones or browser users in any way.

As some games have started to understand, people like different things in games. I like building stuff and optimizing structures, some like carnage, others like to level up or wait 45 minutes for a virtual beef pie to be ready. You cannot have all that in one game if you’re the only content generator.

Also, if the engine is not open, people won’t be able to enhance it for their particular worlds. It doesn’t have to be open source, but it has to have a good API and an efficient plugin system. Tools to create mods and content is also essential, but the real deal is to give the users freedom to create their versions of the game and be able to connect them all.

If every game is also a server, people can create their small worlds inside the bigger world, that is in the central server. A business strategy would be, then, to host those worlds for people that really cared about them. Either host for free in exchange of ads and content generation, or paid hosting for the more serious users. You can also easily sell content, but more importantly, you can create a whole marketplace of content driven by the users! Read Neil Stephenson’s Snow Crash and you know what I mean.

I think Apple and Google have proven over and over that a market with apps generated by the users is very effective indeed! Intel is just following the same path with their new App store, so yes, this is a good business strategy. But, it’s still fun for a wider range of people, from game addicts to casual gamers, from heavy modders to passive Facebook users.

There are many ways of doing that, maybe not all of them will be successful, but at least from my point of view, until that day arrives, no game will be fun.

Post-Agile

A while ago I wrote an article about Agile and Scrum and wanted to write another one following my recent experience with Agile. However, somehow I couldn’t add anything of that great value to my original post that would be worth a new one.

And now I know I don’t have to. In this fantastic post, Gwaredd takes a deep look into all failures and successes of Agile, with the common misconceptions of believers and decision-makers. In the end, the so called “Post Agile”, is just plain common sense.

Ad infinitum

Quality is fundamental in any job, and software is no exception. Although fairly good software is relatively easy to do, really good software is an art that few can truly reach.

While in some places you see a complete lack of understanding about the minimal standards of software development, in others you see it in excess. It’s no good either. In the end, as we all know, the only thing that prevails is common sense. Quality management, all sorts of tests and refactoring is fundamental to the agile development, but being agile doesn’t mean being time-proof.

One might argue that, if you keep on refactoring your code, one day it’ll be perfect. That if you have unit tests, regression tests, usability test (and they’re also being constantly refactored), you won’t be able to revive old bugs. That if you have a team always testing your programs, building a huge infrastructure to assure everything is user proof, users will never get a product they can’t handle. It won’t happen.

It’s like general relativity, the more speed you get, the heavier you become and it gets more difficult to get more speed. Unlike physics, though, there is a clear ceiling to your growth curve, from where you fall rather than stabilize. It’s the moment when you have to let go, take out what you’ve learned and start all over again, probably making the same mistakes and certainly making new ones.

Cost

It’s all about cost analysis. It’s not just money, it’s also about time, passion, hobbies. It’s about what you’re going to show your children when they grow up. You don’t have much time (they grow pretty fast!), so you need to be efficient.

Being efficient is quite different on achieving the best quality possible, and being efficient locally can also be very deceiving. Hacking your way through every problem, unworried about the near future is one way of screwing up things pretty badly, but being agile can lead you to the same places, just over prettier roads.

When the team is bigger than one person, you can’t possibly know everything that is going on. You trust other peoples judgements, you understand things differently and you end up assuming too much about some of the things. Those little things add up to the amount of tests and refactoring you have to run for each and every little change and your system will indubitably cripple up to a halt.

Time

For some, time is money. For me, it’s much more than that. I won’t have time to do everything I want, so I better choose wisely putting all correct weights on the things I love or must do. We’re not alone, nor is all we do for ourselves, so it’s pretty clear that we all want our things to last.

Time, for software, is not a trivial concept. Some good software don’t even get the chance while some really bad things are still being massively used. Take the OS/2 vs. Windows case. But also some good software (or algorithms or protocols) have proven to be much more valuable and stable than anyone ever predicted. Take the IPv4 networking and the Unix operating system (with new clothes nowadays) as examples.

We desperately need to move to IPv6 but there’s a big fear. Some people are advocating for decades now that Unix is already decades deprecated and still it’s by far the best operating system we have available today. Is it really necessary to deprecate Unix? Is hardware really ready to take the best out of micro-kernel written in functional programming languages?

For how long does a software lives, then?

It depends on so many things that it’s impossible to answer that question, but there are some general rules:

  • Is it well written enough to be easy to enhance to users’ request? AND
  • Is it stable enough that won’t drive people away due to constant errors? AND
  • Does it really makes the difference to people’s lives? AND
  • Are people constantly being reminded that your software exists (both intentionally and unintentionally)? AND
  • Isn’t there something else much better? AND
  • Is the community big enough to make migration difficult?

If you answered no to two or more questions, be sure to review your strategy, you might already be loosing users.

There is another path you might find your answers:

  • Is the code so bad that no one (not even its creator) understand it anymore? OR
  • The dependency chain is so unbearably complicated, recursive and fails (or works) sporadically? OR
  • The creator left the company/group and won’t give a blimp to answer your emails? OR
  • You’re relying on closed-source/proprietary libraries/programs/operating systems, or they have no support anymore? OR
  • Your library or operating system has no support anymore?

If you answered yes to two or more questions, be sure to review your strategy, you might already be on a one-way dead-end.

Ad infinitum

One thing is for sure, the only thing that is really unlimited is stupidity. There are some things that are infinite, but limited. Take a sphere, you can walk on a great circle until the end of all universes and you won’t reach the end, but the sphere is limited in radius, thus, size. Things are, ultimately, limited in the number of dimensions they’re unlimited.

Stupidity in unlimitedly unlimited. If the universe really has 10 dimensions, stupidity has 11. Or more. The only thing that will endure, when the last creature of the last planet of the last galaxy is alive is his/her own stupidity. It’ll probably have the chance to propagate itself and the universe for another age, but it won’t.

In software, then, bugs are bound to happen. Bad design has to take part and there will be a time when you have to leave your software rest in peace. Use your time in a more creative way because for you, there is no infinite time or resources. Your children (and other people’s children too) will grow quick and deprecate you.

Calliper, chalks and the axe!

Years ago, when I was still doing physics university in São Paulo, a friend biochemist stated one of the biggest truths about physics: Physicist is the one that measures with a calliper, marks with chalk and cuts with an axe!.

I didn’t get it until I got through some courses that teaches how to use the mathematical tools available, extrapolate to the most infamous case, than expand in a series, take the first argument and prove the theorem. If you get the second argument, you’re doing fine physics (but floating point precision will screw up anyway).

Only recently I’ve learnt that some scientists are really doing a lot by following in the opposite direction. While most molecular dynamics simulation are going to the quantum level, taking ages to get to an averagely reasonable result (by quantum standards), some labs are actually beating them in speed and quality of results by focusing on software optimizations rather than going berzerk on the physical model.

It’s not like the infamous Russian pen (which is a hoax, by the way), it’s only the normal over-engineering that we normally see when people are trying to impress the rest of the world. The Russians themselves can do some pretty dumb simplifications like the cucumber picker or over-engineering like the Screw Drive that, in the end, created more problems than solved.

Very clear, in software development, the situation can be as bad as that. The complexity of over-designed interfaces or over-engineered libraries can render a project useless in a matter of months. Working around would increase the big ball of mud and re-writing from scratch would take a long time, not to mention include more bugs than it solves.

Things that I’ve recently seen as over-engineering were:

  • Immutable objects (as arguments or on polymorphic lists): When you build some objects and feed them to polymorphic immutable lists (when creating a bigger object, for instance) and then need to change that afterwards you have to copy, change and then write back.
    This is not only annoying but is utterly ineffective when the list is big (and thousands of objects need to be copied back and forth). The way out of it is to use the bridge pattern and create several (RW) implementations of your objects and lists and whatever you have but that also increases a lot on code complexity and maintenance.
    My view of the matter is: protect your stuff from other people, not from yourself. As in “Library Consistent” or “Package-wise Consistent”.
  • Abuse of “Standard algorithms“: Ok, one of the important concepts in software quality is the use of standards. I’ve written it myself, over and over. But, like water, using no standards will kill your project the same way as abusing of them.
    So, if you create a std::set that gives you the power of log(N) searches, why the heck you’d use std::find_if ( begin(), end(), MyComparator() );, that gives you linear searches? Worse, that find was actually before each and every insert! std::set guarantees at least N.log(N) speed on insertion, but the “standard fail-safe assurance” was giving it N².log(N). For what? To assure no duplicated entries were ever inserted in the set, what was yet another thing guaranteed by the default container in question.
    All in all, the programmer was only trying to follow the same pattern over the entire code. A noble cause, indeed.

Now, I’m still defining what’s worse: over-engineering or under-engineering… Funny, though, both have very similar effects on our lives…

Serial thinking

I wonder why the human race is so tied up with serial thinking… We are so limited that even when we think in parallel, each parallel line is serial!

What?

Take the universe. Every single particle in the universe know all the rules (not many) that they need to follow. On themselves, the rules are dumb: you have weight, charge and can move freely round the empty space. But join several particles together and they form a complex atom with much more rules (combined from the first ones) that, if combined again form molecules that form macro-molecules that form cells that form organs that form organisms that form societies etc. Each level makes an exponential leap on the number of rules from the previous one.

Than, the stupid humanoid looks at reality and says: “That’s too complex, I’ll do one thing at a time”. That’s complete rubbish! His zillions of cells are doing zillions of different things each, his brain is interconnecting everything at the same time and that’s the only reason he can breathe wee and whistle at the same time.

Now take machines. The industrialization revolutionized the world by putting one thing after the other, Alan Turing revolutionized the world again by putting one cell after the other in the Turing tape. Today’s processors can only think of one thing after the other because of that.

Today you have multi-core processors doing different things but still each one is doing things in serial (Intel’s HyperThreading is inefficiently working in serial). Vector processors like graphic cards and big machines like the old Crays were doing exactly the same thing over a list of different values and Quantum computers will do the same operation over an entangled bunch of qbits (which is quite impressive) but still, all of it is serial thinking!

Optimization of code is to reduce the number of serial steps, parallelization of code is to put smaller sets of serial instructions to work at the same time, even message passing is serial on each node, the same with functional programming, asynchronous communications, everything is serial at some point.

Trying to map today’s programming languages or machines to work at the holographic level (such as the universe) is not only difficult, it’s impossible. The Turing machine is serial by concept, so everything built on top of it will be serial at one point. There must be a new concept of holographic (or fractal) machine, where each part knows all rules but only with volume you can create meaningful results, where code is not done by organizing the high-level rules but by creating a dynamic for the simple rules that will lead to the expected result.

How then?

Such holographic machine would have a few very simple “machine instruction” like “weight of photon is 0x000” or “charge of electron is 1.60217646 × 10^-19” and time will define the dynamics. Functions would be a pre-defined arrangement of basic rules that must be stable, otherwise it’d blow up (like too many protons in the nucleus), but it wouldn’t blow up the universe (as in throw exceptions), it would blow up the group itself and it would become lots of smaller groups, up to the indivisible particle.

The operating system of such machine should take care of the smaller groups and try to keep the groups as big as possible by rearranging them in a stable manner, pretty much as a God would do to its universe when it goes crazy. Programs running on this operating system would be able to use God’s power (GodOS libraries) to manipulate the groups at their own discretion, creating higher beings, able to interact, think and create new things… maybe another machine… maybe another machine able to answer the ultimate question of Life, the Universe and Everything.

I know letting the machine live would be the proper way of doing it but that could take a few billion years or I’ll be quite tired of engineering the machine and it’s OS and I’ll just want to the the job done quickly after that…

Why?

There is a big fuzz about Non-Polynomial time problems (NP-complete), those that can’t be solved in a reasonable (polynomial) time. The classic example is the travelling salesman problem where a salesman has to go to each one of a number of cities. Which is the best path to follow to visit all of them in the smallest distance possible? With 3 or 4 it’s quite simple but when you have lots like 300 it becomes impossible for normal (serial) computers to solve.

Another problem quite fancy is the Steiner tree problem, where you have some points and you want to connect them using the least amount of strings. This is as complex as the problem above, can take forever (longer than the age of the universe) for relatively small sets of points, but if you use water and soap the problem is solved almost instantly.

Of course, soap films cannot calculate the last digit of PI but because every part of it know a small list of basic rules (surface tension increased by the soap molecules derived from opposite charges between atoms) every particle of the machine works together at the same time and the result is only achieved because the dynamic of the system has it’s least energy (least amount of strings) in that state.

It’s true that today’s computers are very efficient on working on a wide range of problems (thanks to Turing proving the classes of problems his tape could solve) but there are some that it can’t, given that we only have a few billion years yet of universe to spare. Such problems could be solved if there was a holographic machine.

UPDATE:

More or less what I said was practically applied here. Thanks André for the link, this video is great!

Do you know a place…

… where developers can use the full power of their creativity to produce only 10% of good stuff (good enough to pay for the remaining 90% or more)?

… where politics is left to the politicians? Where money if left to the money-makers? Where personal marketing, boot licking and gossip are not encouraged nor accepted?

… that values work and creativity much more than media and publicity?

… that invests in new ideas, where people are not afraid of the certain failure most of them will lead? Where investors understand that ideas are a high risk investment rather than a 1 year payback savings?

If you answered ‘yes’ to at least one of the questions above, you’re a lucky guy/gal!

Why are we so lame?

It has now passed almost 170 years since the first programmer did the first code in the first analytical machine and yet we are so lame I can barely be proud of what I do, professionally of course.

Coders are now among the richest people in the world, programs sell more than bananas nowadays and the investment in technology is considered now one of the basic blocks of modern society and still we can’t build a piece of software that lasts for more than 5 years without being completely scraped.

The open source community is probably the biggest anarchist movement in history with all the good and bad things we all knew about anarchy and for the surprise of capitalists it’s working far better than big technology companies (check IBM’s OSS support and Vista’s problems in the general news).

Of the hundreds of open source softwares that emerge only a very small fraction have success but yet they manage to stay for longer in a stable status and growing smoother than paid software.

Companies are so worried about money that even tech companies can’t write proper code. Scientific institutes are so worried about doing the perfect way every time that they re-write everything everytime and still need to re-write from scratch next time again. Money and ego are the villains in our industry and unfortunately most of us are affected by at least one of them.

Case studies

Banks, for instance, focus on what works instead of what will last. The result is a complete mess (our beloved ball of mud) that eventually have to be re-written in the future and replaced by another ball of mud with bits of the legacy balls (just in case something breaks). If customers could see the code that run a bank before opening an account they will never open it, in ANY bank.

Internet companies have the same problem but a few more, it must look good and it must be new. It doesn’t matter if the code is good, or if someone really need that feature, but the more features it have the better. (Not) Maintaining that in the future is a completely different problem.

Big companies only follow standards when it’s not needed, like writing core libraries in PHP because that’s the company’s default language or sticking crappy systems together with tape just because they’re the standard libraries used worldwide.

Scientific institutes are not free of problems too. Money is much less a problem but the ego is so big that it can be even worse. Everyone wants to re-write everything their own way and no one disagrees and just shake their heads and wait years to have another non-working beta system to maintain alongside with all other systems made by people that left the institute decades ago but there is still one person using (and normally afraid of moving as it’s not his code as well).

So, why?!

Simple: ego, fear, pride and prejudice.

Not many programmers really care about money, as much as they care about their own ego. The money problems in private companies is institutional and not personal. Our problem is personal.

Most CTOs, CIOs etc know about all problems of taking the actions they take but they’re often afraid of taking radical decisions (even when it’s for the better) or their ego demand them to change something that is working well just because it was someone else who did it.

Most programmers will have a huge pride of their systems and consequently an even bigger prejudice on the others’. Reason can’t work where pride and prejudice rules, nothing good can come out from an environment as bad as that.

Now take those four problems and you know why are we SO lame!

This is also the same reason why open source software is not as lame as closed source. Programmers still have ego and pride but code is pure logic. When your code is exposed you won’t be proud of a bug and if someone points you to one in your code you rather fix it than yell at him/her. In this case your ego and pride will not save your job but make you loose credibility in the community and your project will be put aside and other will arise, so if you don’t cooperate, you die.

Open source promotes dialogues, refining code and algorithms, finding bugs and fixing them the best way possible. It might be (arguably) slower than big companies but still the overall local quality is much better.

What to do?

It’s not just a matter of solving one or other problem, as the environment is already filled with rubbish up to the top, it’s a matter of acting different, from scratch. (my ego is working now)

If you know something is wrong, report it. If someone says it’s impossible to do, demand proof. If someone takes actions based on ego, fear, pride or prejudice, open your mouth and speak loud.

You might loose your job but I’d rather loose my job than work on such a lame place and today, unfortunately, it quite difficult to find somewhere not lame.

Document to understand

Not always you have the opportunity to write a fresh new code. Sometimes you have to face a huge codebase, be it either one big monolithic code or thousands of small scripts and programs, it doesn’t matter, it’ll definitely be a nightmare. So, how to avoid stress and go through it with the fewest scratches possible?

It all depends on what you can do with the code…

If the last programmers were nice to you they have prepared test cases, documentation, doxygen comments, in-line comments, a wiki with all steps to compile, test and use the software and a plethora of resolved (and explained) problem-cases that can happen. I’m still to see that happening…

As Kernighan exposes in ;login: (apr 2006) writing tests should be the primary task of the programmer and not code. With existent codes, documentation should be the primary task instead of direct changes to the program.

If the program is documented already, your first task should read both the docs and the code, side by side but the case might probably be that you won’t understand properly. Nothing wrong with you, just that programmers tend to document what they don’t understand and fail to do so for what they do understand. But if you start changing old docs you might end up screwing everything so here’s a quick tip:

  1. Copy the code to a temp area so you can play at will
  2. Compile the code with your mind, read the code, understand from where all variables are coming, go backwards and check and add a comment before each line saying what each important line does
  3. Group lines in domains and create a bigger comment. This is specially important for balls of mud where the whole code is in a single function
  4. Check against the old docs and see if what it’s written is actually what’s happening. Documentation tend to be out of date very quickly

Steps

One side effect of documenting is that eventually you’ll understand the code better than the original coders. I explain. The original coders had one objective in mind: fix the bug. Most of the time and specially with balls of mud the fixes tend to be dirty, written as temporary and run forever. The original coder, most of the time, didn’t know how that bugfix fitted on the whole, he/she just knew it worked.

You, on the other hand, know the whole in a way no other coder had know before because you have documented it from start to finish and have the knowledge of what’s at stake on each change.

But that power have a problem: on the day you stop documenting you’ll be the “old coder” and stop understanding the code as a whole and start writing bad bugfixes yourself! So be diligent, be patient and most of all loyal to your grater purpose: to code better.

Back to the bug fixes… Once you find them you’ll notice that most of them are useless or redundant, or that lots of them can be replaced by a very small and simple shell script. Once you learnt all steps of a program you visualize the flow and it becomes very clear all shortcuts and optimizations available to you, and you have now the power to change it.

But after all those changes, a good chunk of the documentation is useless, specially those you made so, why bother? I tell you why, if you hadn’t documented in the first place you wouldn’t be able to optimize it that deep and throwing away most of your past documentation will (should) encourage you to document it again, in a higher level. Doing it you’ll see things that you couldn’t before (even when you had the whole system in your mind) because the system was not organized! That’s the next step!

Every time you document everything and see the whole system in your mind you visualize all optimizations available, this will force you to throw away old code and docs and redo again, which in turn, will come back to the same point but one step further.

The pure act of documenting allow you to optimize. Throwing away old docs allows you to go one step further.

Documents in tree

Instead of throwing away old docs might not be a good idea because people will not understand why you optimized that way and will probably go back to the original code once you’re off. Keep all documents, organize them in a tree and show only the high level docs at first, if one wants to know more it can go deeper in the tree to find out how it was and why you made it that way.

It’ll also help you to understand your own optimizations and change in the future when it makes no sense any more.

Wasting time

Some will argue you’re wasting time documenting when you could understand the code just by looking at it (use the source, luke) but that just isn’t true. People, as well as computer, have a short term and a long term memory and most codes won’t fit entirely on your short term memory (if they do, get a better job).

Navigating through your long term memory is expensive and will switch context on your short term memory and you’ll not be able to make connections properly. But if you have everything written in an organized structure it’ll be much easier for your brain (or any machine) to infer about the code.

It’s the same as testing, you may spend half of your project’s time doing the test cases but that could (and probably will) save you hundreds of hours (and probably your job) if something goes wrong, and it’s very likely to happen.

At last, writing test cases, documentation and proper code is not only part of your job, it’s part of your grater purpose in life: to code better. No job in the world is worth writing bad code, it’s like lying to keep the job… just don’t do it.