Dan Kaminsky's Blog

Executive Summary

Open Source Software has proven itself to be inordinately successful
in providing high quality software in a method that, on its face,
defies standard business reasoning. This article clarifies OSS in the
language of economics such that the true reasons behind its success
may be understood by everyone, not just the coders. The following
summary, written by Jack Carrol, well explains the contents of this
essay:

Open source software development is a barter economy with some very
unusual economic properties. The original author of a piece of
OSS gets paid, but not in money. The author gets paid in
software. So does everybody else who contributes a feature or a
bug fix. This form of payback is very real, for all of its
intangibility. The return is huge — everybody who contributes to
the growing program ends up with the use of a much more capable
and higher-quality product than they could afford to develop on
their own, and they get a product which has been carefully
enhanced to meet their exact needs. The actual developers aren’t
the only contributors and beneficiaries, either. Everyone who
contributes documentation, distribution services, mutual tech
support, or publicity participates in this barter economy.

Some other highlights:

• Source Licenses, where businesses pay to receive access to the
  codebase, are a far different creature than Open Source Software,
  as they end up placing fewer eyeballs against more bugs–in
  essence, it devotes less resources to more weaknesses.

• OSS means outside engineers see your code before it crashes, while
  it remains in the development cycle–and these engineers are
  likely to assist in the fixing of your bugs, saving you
  substantial amounts of money.

• Closed source code development forces sites that require custom
  features to choose between software that is too expensive to
  produce or code that is too unstable to trust, with significant
  diversion from core competencies unavoidable due to the massive
  redudant effort that is necessary for any complex project.

• Closed Source software providers are actually less likely to
  provide timely code fixes due to their organizational structure,
  and the “insurance premium” that managers justify paying out large
  amounts of money to a single supplier with actually ends up
  creating a support monopoly, in the sense of the supplier knows
  that it´d be too expensive for you to leave and absolutely
  impossible to find anyone else who could patch the code.
  LinuxCare can be founded, MicrosoftCare never can be.

• OSS merges the advantages of custom code with the advantages of
  Commercial Off-The-Shelf(COTS) products, essentially creating
  Modifiable COTS, minimizing the amount of inefficient labor that
  must be expended to create a product.

• OSS allows hiring managers to truly understand the skills of an
  individual who has coded under an Open Source license, thus
  increasing both his hirability and his salary range–OSS
  developers, quite frankly, are more bankable, especially since
  they have the ability to prove themselves worthy of any position,
  not just one that fits the position they´ve taken at a given
  company.

• Specialization, and Ricardo´s Theory of Comparative Advantage
  provide significant backing behind a free and open code
  architecture.

• When one examines the rest of our society, it quickly becomes
  clear that when stability matters, openness is the only way to
  achieve it. Stability matters for business. Period.

Introduction

Open Source Software works. A simple statement, but finally
understood–code developed openly, without significant restrictions on
redistribution, works better than code developed under almost any
other software methodology. Business is finally beginning to accept
us on our technical merits, although it still doesn´t yet understand
why we´d work so hard on something we just don´t seem to be getting
paid for. They know what we´ve done, just not why we´ve done it or
how we´ve managed to do it so effectively.
We have proved our paradigm–and considering how different it is from
“established” software business practices, it truly is a
paradigm–successful, and like any success story, there are many in
the world who are desperate to understand why, so that they too may
follow our road to success. They still have their doubts, and
contrary to what some of us might think, they have good reason to
doubt. Many in the industry we would drastically change have honest,
good, and valid reasons for being doubtful of the seemingly siren call
of Open Source. It is not enough for them to understand the empirical
reality that OSS is blisteringly, frighteningly, and tremendously
successful in its pursuit of speed, stability, quality, and soon,
ubiquity. They must know why, in terms they understand–and, since it
is always valuable to understand the perspective of users, so must we.

The Ghost Of Source Code Past

One of the primary reasons for experienced members of the IT community
posessing a degree of wariness surrounding Open Source software is
that it seems like nothing more than the old promise of source
licenses. For a very long period in computing, publishers made the
source code to all applications commonly available and often licensed
it to large businesses for mission critical applications. Such
licensing caused serious problems if the business hired programmers to
add the features they required to the code–how were the original
authors supposed to support something they didn´t write? Needless to
say, an environment that is difficult or impossible to support is
difficult or impossible to keep reliable, so businesses were forced to
forgo any site-specific code patches and only use the licensed code
for reference purposes.

This, of course, wasn´t always businesses licensed the code for in the
first place. Some had been sold on the idea of maximum flexibility
and maximum reliability through the ability to hire programmers to
customize everything down to the core. But the empirical reality was
that once a site began making such modifications, they were required
to hire additional programmers to help support something that nobody
else could have the means to understand. The final results were so
ugly–massive buck-passing, unacceptable downtimes–that the
predictable and supportable comfort of closed source software simply
became the new status quo.

This, of course, is the oft-ignored voice of experience´s Tulip Claim
against Open Source Software: We´ve had access to the source before,
and it made things worse. Why, then, should “Open Source” be any
different?

The short answer: Open Source means outside engineers see your code
before it crashes–and can do something about it.

The Evolution of Access

Linus Torvalds, creator of the classic Open Source success story known
nigh-eponymously as Linux, is known also for what Eric S. Raymond
named Linus´s Law of Software Engineering: “Given enough eyeballs,
all bugs are shallow.” The inverse often also holds true–“Given few
enough eyeballs, all bugs are deep.”
Linus´s law explains the false promise of the source licensees.
Having spent so much money obtaining the code they were to modify,
pressure existed to keep the code out of as many eyes as possible.
Exposing too much to the original authors might lead them to rip off
company code and put it in the next version, so formal or informal
limits to the lines of code that could be transferred were imposed.
Worse off, even these limited code transfers would only occur if a
site got desperate enough to actually seek external support. Source
licensees were faced with the problem of fewer eyeballs facing more
bugs–a recipe for disaster if there ever was one.
Open source works far differently. While both methods grant access to
the source code, open source demands that all code written be freely
available, with limitations on the distribution of modified code
forbidden. This seems ludicrous from a business perspective until one
realizes the importance of sticking to one´s core competency. An
architecture firm is not a software firm, and neither is a clothing
store, but both can and do have needs for custom code that fits their
needs, works correctly, and can be supported to the required degree.
Engineering software in-house, either from scratch or by use of an
costly license, requires expensive programmers to execute redundant
labor to write inadequately tested and difficult, if not impossible to
support code,. Worse, it requires businesses to divert more and more
of their resources away from their core competencies as their systems
remain unreliable. Outsourcing code saves much of the hassles of
hiring and maintaining programmers, but brings in its own host of
problems beyond the management tax while addressing few of the quality
issues that any closed source development will incur. The fact is,
testing and support requires manpower that is doubtful to be
significantly less expensive for an outsourcing company vs. an in
house department, meaning cheap testing is cheap testing no matter
where you slice it. It is no wonder that closed source software
producers enjoy this paradigm–there´s simply so much more control for
the original authors when customers can´t put their own teams together
to implement a feature or fix a bug they desperately need fixed.

The Best Code Money Can´t Buy

Large corporations tend to believe that their payments lead to
software companies “looking out for them”–and, by corrolary, that
free code is completely free of obligation and quality. Professor
Manoochehr Ghiassi, chair of the department of Operation and
Management of Information Systems at Santa Clara University, explains:

Managers would prefer to pay a company an “insurance premium” so
that they may interact with a corporate support structure
organized in a way they are accustomed to. This is why,
historically, the Santa Cruz Operation(SCO) would win clients
over the far less expensive BSD based systems.

It is therefore sadly ironic that most closed source software
companies refuse to address security issues in their software until
public exploits for those security flaws are released. Time and time
again, we learn of an attack mailed to the programmers weeks before
the exploit release, and time and time again we can only see something
done about it some days(or weeks) after the vulnerability is
exploited–in other words, nothing is done until somebody loses time,
money, and productivity from a flaw. Dr. Ghiassi explains why this
is:

Why do you think that [companies] delays the distribution of
patches for so long? Writing code is the least expensive
part of any fix. Distributing the new code to each customer
is traditionally so expensive that delaying the release of
patches until the next revision of the software is the only
affordable solution.

A further issue, mentioned by Red Hat Software, is that quickly
releasing new versions leads to resellers demanding their now-obsolete
unsold product be exchanged for the newest release. Microsoft´s
inclusion of the Critical Update functionality, which automatically
checks the Internet for “important fixes” for Windows 98 is a
development to be cautiously lauded for solving this very problem.
(I say “cautiously”, since the “trojan patch” that simultaneously
attacks all machines is much simpler to execute when everybody patches
simultaneously.) But, even with technological advances, Microsoft and
most other companies can only justify crisis-speed responses after,
and not before, the occurance of a crisis. The problem is not
technical, just organizational. Someone must beg before they will
respond.

This behavior is not surprising, given the minimal harm a client can
do to their bottom line–this is especially true in the case of
Microsoft. Large companies may want someone to sue, but amazingly
software companies are practically untouchable. These closed source
suppliers are legally absolved of any responsibility due to claims of
artistic right–basically, software is like speech, and to mandate
correctness would be akin mandating perfect grammar. These are valid
claims, mind you, until the industry turns around and succumbs to the
urge to patent such “art” suspiciously like a manufacturered product
design. (This does little to enhance credibility and will probably
eventually backfire in the face of the software industry. It´s not
too late…yet.) As things stand, only the moral and technical
consciences of programmers can supply a pre-emptive strike against
insecurity.

Unsurprisingly, we only see such moral and technical consciences
exhibiting themselves in the open source world, where code can freely
travel from programmer to programmer. This ought to ring strange to
businessmen–why would the uncompensated organization or individual be
so much more likely to operate in the best interests of its clientele
as compared to their compensated counterparts? There are a couple
reasons.

Support Monopolies are a major cause of the existing situation. The
closed source software paradigm gives users the choice of either
putting up with whatever the original authors provide or ceasing usage
of their software. Since the cost of switching from an installed
application can be substantial, businesses are forced to accept the
whims of a monopoly agency that “holds all the cards”, to borrow a
cliche. Eric S. Raymond, author of the industry classic The Cathedral
And The Bazaar, euphemizes this relationship as “a bad strategic
business risk”. Whatever you call it, the effects shouldn´t be
surprising–most monopolies behave with such disdain for their
customers. The basis of the monopoly is that the authoring company
under the closed source paradigm is the only company that can both see
how the code really works and fix any problems. By switching to Open
Source, the authoring organization still possesses a competitive
advantage in providing support, but other individuals and
organizations can still learn the platform and reduce the end user
cost of receiving support. Linux, unlike the free Unixen of the past,
has been able to parlay inexpensive computers and inexpensive network
access into an amorphous but quite real user support base. Extremely
technical questions are routinely answered, whereas many closed source
outlets make it difficult to even find a way to submit bug
reports–perhaps they believe such a link might constitute an
admission that their software has bugs. Open source can be so
successful providing support with no backing organizational structure
since the large number of individual specialists are all available to
answer those questions which are comparatively simple for them to
answer. This support environment is self-scaling–the more users come
in, the more users learn and are able to support eachother. However,
many businesses do not feel comfortable dealing with such an
undefinable support architecture–unlike closed source, these
businesses can choose alternate support providers while still using
the same software. LinuxCare is slated to handle the support needs
for Dell, and IBM is also building Linux support systems. There can
never be competition in the market for support of a closed source
architecture–there could never be a MicrosoftCare–but open source
can create support that lasts as long as there is demand, even if the
original company goes out of business. It should be painfully obvious
which platform truly provides better opportunities for support.
Open Source´s reach goes beyond resolving the support monopoly.
According to Steve Rostdedt of Lockheed-Martin, each business often
has to choose: Either develop code that specifically fits the required
specifications, or purchase code that is COTS, for Commerical Off The
Shelf. Once purchased, the overall system is “hacked” such that the
(very inexpensive) COTS solution can be used in place of its
customized and very expensive replacement. Needless to say, there are
definite problems with both approach. While custom code will function
exactly according to specification, it can only do so at great cost to
stability and the bottom line. COTS is cheaper and ostensibly more
stable in a general environment, but often requires significant
modification to systems surrounding it. In fact, these modifications
often end up costing as much and reducing stability equal to the
amount supposedly saved by going COTS–Steve tells stories of even the
most staunchest COTS advocates scaling back, simply because they got
burned by this so badly. The interesting thing, though, about Open
Source is that it´s essentially Modifable COTS–a company can take
advantage of software that does 90% of what´s needed and literally add
in the remaining 10% without either starting from scratch or making do
with out it. In that way, Open Source can absorb the best advantages
of custom code while leveraging the economies of scale that COTS
provides.

But how has Linux, through Open Source, managed to advance its code to
the impressive state it stands at today? Linus´s Law, once again,
states that with enough eyeballs, all bugs are shallow. The Open
Source paradigm increases the supply of eyeballs available to scan and
satiate the demand for stable and effective software. Interestingly,
both the quantity of buggy and/or suboptimal code replaced is
increased as the price per fix approaches zero.
Zero? People can not live on zero dollars, nor can businesses survive
on zero revenues. How can zero cost software make economic sense to
anybody involved, except for those who receive it? In short: Trick
question. The software isn´t zero cost, and even those who write the
initial code receive update code in return. Consider: Bad code
equals more work, while good code doesn´t. Badly written code must
be repeatedly fixed, hacked around, modified, tweaked, ruled out, and,
when the eventual upgrade occurs, checked extensively for “useful
bugs” that the rest of the codebase depends on. The longer bad code
goes undetected, and the longer it remains in a codebase, the higher
the expense it creates in both time and money. The Open Source
process decreases the detection period and shortens the amount of time
and energy necessary for a corrective patch to be
written–essentially, OSS removes much of the expense of bad code.
There is another benefit of the time compression that should not go
unnoticed. All code written for corporate usage goes through a
limited development period before it is sent to production. It is in
this period that it is, by far, most cost effective to find bugs and
patch them. It stands to reason that the only thing better than bugs
being found before they can cause a problem is bugs being found before
they´re put in place to create one. Open Source´s unrelenting nature
of code improvements is the only process known that can affordably do
this. Finally, additional patches of necessary features are publically
submitted, thus increasing the amount of useful code in the codebase.
The sheer existance of Linux, with an estimated R&D expense of over
two billion dollars covered entirely by volunteer labor, is proof of
the viability of this argumentation. In essence, since the expense of
developing software without OSS is thus higher than the cost of
developing it in an open environment, OSS must be credited financially
with reducing costs–in a sense, the process pays for itself in speed
of quality development.

In the final analysis, business requires stable software. No amount
of money can outweigh the systemic and malignant effects of downtime
of corporate computers–everything from morale to public relations to
bottom line income and productivity are slashed when the tools that
the workforce depends on just aren´t there. The GNU Development
Platform with which most Open Source software is written has finally
reached the level of maturity that not only acceptable but
best-of-breed solutions may be built with it. Apache, the open source
web serving software, does not hold a commanding majority market share
against the Microsoft marketing juggernaut merely because it´s free.
Linux, the open source poster child, has not been the only Unix to see
its share grow simply because of its Penguin mascot. Linux and Apache
are stable under extreme load. If you accept the argument that
downtime is extraordinarily taxing against the financial and human
resources of any company, one must account the prevention of these
downtimes as a dollar value win for the company. Open Source is loss
prevention, pure and simple.

All Things Large And Small

Yet, while significant value is delivered to the company, what about
the individual? Frankly, most of the stability concerns of the large
company simply are not applicable to one person. If my computer goes
down, I reboot it. Annoying, but it´s doubtful to affect third
quarter profits. What economic rationale do I have to produce, debug,
or support Open Source software? The simplest answer is that it
directly affects my hirability and salary range. Human resources
people don´t have much to work with under the closed source
paradigm–A recommendation letter from a job the applicant no longer
works at, claims of business duties verifiable only by asking a
reference who can be sued for excessively contradicting the applicant,
and an interview process that penalizes the meek and rewards the
liars. Worse, anything that smacks of actual examples of what the
applicant actually wrote at the previous place of employment risks
serfdom-suits over the divulging of trade secrets. Open Source gives
companies much more to work with. Instead of claiming, applicants can
prove their skills providing support, can show their own source code,
can provide evidence of their contributions to large software
projects. This makes OSS developers more bankable, and thus worth
more on the pay scale.

What is good for the developer can be very good for the development
house. Untold millions have been made from designing web sites that
run off of Unix and Apache. If stable code is key, the ability to
prove one´s prowess using it–free or not–is extremely valuable. A
company that not only develops sites with the code but also develops
the code to its fullest degree strengthens its online presence
considerably as the new code gains the appreciation and respect of the
net community. This “saves” potential clients the trouble of worrying
whether a certain supplier is a possible scam artist and paves the way
for the organization.

Therein lies the answer to the question James Sullivan posed to me.
“Suppose,” he began, “suppose you could earn one hundred thousand
dollars if you published your essay in a magazine. Would you still
release it freely?” I paced for a moment, and replied, “If I lived in
a world where my essay had a value of a hundred grand, then open
source means that I´d be reading millions of dollars worth of the
essays of others, and all parties involved would be rich.”
There´s a more subtle advantage for the individual as well. While one
doesn´t have tremendous influence over what projects one is placed on
in a corporation, the entire range of code, from graphical
manipulation to network file serving to device drivers are open for
the individual to improve in the Open Source landscape. Such
availablity of resources allows individuals to experiment without
risk, express without changing jobs, and specialize in whatever best
matches his or her expanding skillset. Economic theory is greatly
respectful of specialization as it is the means by which resources are
best allocated. Ricardo extended specialization into the
international realm in his theory of comparative advantage.
Comparative Advantage holds that the global economy benefits most when
each state produces that which it generates with the greatest
efficiency. In the case of software, the goods are various tools and
applications, the states are individual programmers, teams of
programmers, or entire conglomerates, and the barriers are licensing
policies that prevent access, modification, or redistribution of the
original source code. While an organization may hold the ability to
write certain code with the greatest level of efficiency, testing or
debugging procedures may be comparatively less expensive by other
programmers. Without large amounts of money providing incentive for
otherwise suboptimal code to be used, the entire system shifts into a
meritocracy, with each programmer and each organization providing the
services they are most comparatively able to achieve. Just as Ricardo
would predict, this minimizes the cost of designing software that
otherwise may never have been created, thus maximizing the value of
code at minimal cost.

Interestingly enough, though, most people who code Open Source
Software have never heard nor thought of much of the previous
argumentation, although there usually is a vague realization that
somehow the code might get them a job. The true reasons many people
work on Open Source Software are non-monetary, but assuredly not
without reward. Some people patch open source code for the same
reason others replace the shocks in a classic car. Others help
support a widely used application because it was given to them freely
and they receive human appreciation for their assistance. Finally,
some people start their own projects because they need their computers
to do something that they just won´t do yet. For a sense of progress,
appreciation, or usefulness, people will circumvent the intermediary
of money if the cost is low enough and the reward is great enough.
However, open source development need not be unpaid–a single sponsor
can wholly subsidize the full time professional activities of an OSS
developer, and systemically the price per unit delivered of each bug
fix will remain negligibly above zero. However, both the sponsor and
the programmer will receive supplementary payments in the form of
public appreciation, mindshare, good will, and perhaps influence in
the future design of the whole product. In other words, the sponsor
receives exactly what advertising is supposed to buy it by literally
handing users something that cost the sponsor little but contributes
significantly to the fulfillment of the target´s desires. It is
advertising canon that targets are most likely to respond positively
once they´ve been provided something desirable by the advertiser, and
thus the cost expended by the sponsor in subsidizing the OSS software
becomes money well spent.

There are further reasons why OSS ends up being more psychologically
worthwhile. When one writes a patch at a large software company, the
credit for that patch is eternally buried within the logs, kept only
in case that code ended up flawed so that person can be disciplined
accordingly. The overall corporate brand stays strong, but the
individual is left having patched something “that shouldn´t have been
wrong in the first place.” The collaborative nature of Open Source is
far different–a useful patch can often result in a appreciative email
from the lead coder, and credit is proudly placed in the announcements
of another version. The power of this should not be taken
lightly–economic theory is very clear that money is only as good as
what it buys, and psychology tells us that appreciation and
recognition are arguably primary ends that individuals will invest
significant energy to achieve.

Finally, the “self-scaling” nature of open source support also applies
to open source development, as the more valuable a certain nascent
project is, the more programmers will attempt to join. It is
estimated only five to ten percent of the Linux kernel remains written
by Linus Torvalds–It is a good thing to end up with code better than
one could create by ones self! This additional support, plus the lack
of a boss demanding x hours of work, mean an individual does not need
to devote himself full time in order to be useful, this can
occur–another advantage over closed source where an individual must
devote themselves to a single organizational position in entirety.
It is the demands of the closed source paradigm that provide the
necessary contrast to understand the critical advantage of Open
Source: Freedom, with a low barrier to entry. The Internet, as it
exists now, almost completely eliminates the marginal costs of code
development by facilitating discussion and data transmission among the
various members of a development group. With an open operating
system, open development tools, open networking subsystems, etc., new
programmers do not need to spend anything more than their time to add
value to the overall system. The importance of this can not be
understated. It´s an economic truth that markets that are difficult
to enter are almost always overpriced and underserved. Markets where
competition is possible almost always create the best solutions–even
Open Source developers compete for who can write the most elegant
patch to a newly discovered flaw. There´s a good reason the computer
industry has always favored open platforms over closed –witness the
success of the entire x86 platform from serial port to PCI bus, the
proliferation of HTML(which allows source code viewing of any and
every page), etc. Openness works–it always has.

The Real Status Quo

Professor Larry Iannaconne of Santa Clara Univeristy writes:

[Open Source Software] requires some very special features of
production and distribution (e.g., marginal costs of
distribution are nearly zero, so that the socially optimal
“price” is zero). On the production side, costs are
substantial, but can be spread among numerous people–indeed,
the comparative advantage argument may imply that costs
should be so spread. The analogous activity is academic
research, pursued with a relative high degree of
collaboration and interaction, performed for little or no
direct money pay, rewarded primarily through reputation, and
(key) performed in tandem with complementary paid work
(teaching and administration).

Such is the life of the programmer who decides to set out and publish
code under the GNU Public License. The dirty little secret, it
appears, isn´t so much that openness is the Hot New Thing but that
openness has always been the standard. Consider–it is very expensive
for a business´s headquarters to physically collapse, therefore
blueprints are subject to public inspection and actually must conform
to many regulations such that the building remains safe. Medicine too
demands the rigorous trials of peer review to determine whether or not
a certain procedure can be considered safe. Indeed, to protect our
freedom, we demand of our government open laws, open debate of those
laws, and open access to the entire political system.

Our education, workplaces, our lives, and our liberty all exist under
the penumbra of Open Source. To deny the value of openness is to deny
those protections we take for granted on a daily basis. The burden,
if you will, is truly shifted to those who would keep their code
secret in an attempt to aggrandize their own company at the expense of
their customers to prove the value in their actions.