Tuesday, December 23, 2008

The trouble with budget surpluses

The trouble with a government (or any big organization) running a budget surplus and sitting on a big pile of cash is that people start asking questions like, "Why don't we do something with that money, especially since there's so much to fix?" At that scale, having some cash ready for a rainy day is not an excuse that people are able to handle, especially when the quoted figure is in the billions of dollars. Surpluses may seem like a lot of money, although when divided on a per-person basis, they seldom amount to much at all.

In this deficit-infested time, it's easy to rail against profligacy in our budgets. I'm not trying to engage in what-ifs here. I'm suggesting a way we can avoid this same problem in the future.

We've been told that it's wise to save up for the future, to have some cash on hand just in case — folk wisdom that was reinforced by the big collapses this year. We also know that it's hard for most people to hear about a surplus in the billions of dollars without wanting to spend it on something.

If an even moderately sized government were wise and careful in its spending, it would quickly accumulate a large surplus. That large surplus would then get people dreaming about capital improvements, social services, and other goodies that drain the public purse.

Those who advocate for smaller governments would say that the problem could be solved by striking the problem at the root — shrinking the government and therefore its potential to accumulate large sums of money. But pools of large capital certainly have their advantages, and whether by conscious choice or mere political expediency a large government must stay large, it could do better to quote the figures based on the population of the governed; that is, on a per-person basis. This might make billion-dollar surpluses less distasteful to the vast majority of us, whose visceral reaction to large dollar amounts placed before us is to spend imprudently.

Thursday, December 11, 2008

Small business owners: don't be a jerk

My girlfriend Sophia is an assistant manager at Abercrombie and Fitch, and is thus bombarded with more than her fair share of rude customers. One story she told me this week was about a woman who went ballistic after asking to try on one of the mannequins' jackets and being told no.

Now, people are just crazy, and Sophia has told me many stories like this before. What was so different this time?

Well, the crazy lady played the business owner card: if it were her store, she would have gladly taken the jackets off the display mannequins. This know-it-all "business owner" then proceeded to hound Sophia for her full name and pressed her for her employee ID so she could file a formal complaint, and refused to go through the normal channels.

It is precisely this self-serving arrogance and provincial, narrow-minded ignorance that keeps small business owners from being taken seriously. As a former small business owner myself, I know that the burden is heavy: you've got to worry about employees, customers and growing your business. On top of that, you have the responsibility to make sure any legal paperwork is in order and that taxes are taken care of. But just because you're able to handle this does not mean that you know all there is to know, and that your way of doing things is the only way.

In response to this growing sense of self-importance, here are three things to keep in mind. (I use these reminders to keep myself in check, too.)

a) Rules and processes have a place, even if you choose to forgo them. As a small business owner, you can get by with fewer rules and processes in place. In fact, in most cases you do much better when you're flexible. But larger businesses have a much harder time being flexible; it's not impossible, just much harder. They have to manage everything more strictly in order to hold together the larger whole. A little sloppiness in your store can be passed off as "charming." In a national chain where customers expect extreme tidiness and consistency, that sloppiness is not charm. It is chaotic, and it is poor business.

b) Not everyone enjoys the same latitude to call the shots as you do. You may be your own boss, but most people have someone else to answer to. I've found that being a business owner, seeing the bigger picture, and having the power to remedy things has turbocharged my ability to take the initiative, even after going back to working for someone else. Still, despite having passion for my line of work and understanding its larger implications, I have much less scope to make important decisions. In large companies, even CEOs don't wield absolute power, because they have a board of directors and shareholders to please.

c) You are not special, so don't expect special treatment. A couple of years ago, I received a parking ticket by mistake. I knew that I had moved my car in time, and so I decided that I would write in to contest it. One of my co-owners suggested that I take a tough stance and mention that I was a business owner — as if that had anything to do with my guilt or innocence. I mentioned it anyway, thinking that a little reminder about my contribution to the community wouldn't hurt. Still, I didn't want to rely on that mostly irrelevant fact, so I put much more effort into stating the facts of the case. I drew a diagram of where I had parked, when I had moved my car, when I had been ticketed, and why it was a mistake. In the end I got the ticket waived, but I have a good hunch it had more to do with stating the facts than mentioning that I was a "business owner."

Saturday, December 6, 2008

Conceptual tools for allocating finite resources

I have lived and worked in Greater Los Angeles for over a year now. Since the area is one of the world's major population centers, it only makes sense that I'd come across plenty of opportunities to observe the interplay between two forces: lots of people and limited resources.

It's a common problem, but always a multifaceted one. Coming up with a solution means that planners (who could be church volunteers, business managers, or software engineers) must consider various factors. Having to stop and think about this can slow down decision making, and even when there's time to ponder and plan, thinking of a solution from scratch is more error-prone than referring to a proven set of guidelines and rules.

Consider the situation when lots of people are elbowing each other to buy concert tickets. There's no increasing the supply of these tickets since a concert hall or stadium can only hold a set amount of people without the Los Angeles Fire Department throwing a hissy fit.

Many eager economics students and libertarians automatically want to apply the law of supply and demand here: why not just raise the prices? In fact, that's what some organizations have done: Britney Spears concert tickets for Staples Center start at $150.00. (How do I know this? My girlfriend told me. Really.)

As the purveyor of tickets, you can do that when you have some idea of how much demand there will be. But what if demand shoots up while your supply stays the same? When it looks like you're about to sell all your inventory and are hours away from turning away your customers and disappointing them, you could double your prices. But then, your customers would be angry that you're gouging them. Your company would appear disreputable for "arbitrarily" changing prices on consumers.

If you're looking at it from an economist's point of view, there's no problem there: you're just changing the price to meet demand. But from a public relations vantage point, you run a very real risk of alienating your customers. It'll look like you're exploiting them, when all you're doing is ensuring availability (while making a few extra bucks).

On top of that potential PR nightmare, there's a downside that even economists must acknowledge. It's the phenomenon that economists call "sacrificing equity for the sake of efficiency." That's all fine and high-sounding, but what does that mean? It means, "It's not fair." And the customer will feel this way. Your costs haven't changed: it still costs you the same to make it, doesn't it? This gives your customers a reason not to trust you, and therefore not to come back.

Changing your prices is certainly an effective tool to curb or increase demand, but raising them is an extremely delicate matter in the age of consumers who expect posted prices to stay the same.

Waiting list: first come, first served
A common alternative to raising prices is to let people buy stuff on a first come, first served basis. The folks at Ticketmaster do this, in case the $150.00 concert tickets sell out (which they often do).

As you may recall from your economics class, this is considered more equitable, but not as efficient: the people selling tickets could make a lot more money just by raising the prices. (If you do not recall from your economics class, shame on you for not paying attention. But hey, now you know. Trust me on this one.)

As far as fairness is concerned, lower prices with waiting lists are an improvement over simply raising the price. But if you're not one of the first served, you probably won't feel like it was very fair. Maybe you had to be at work that time of day, or don't have time to wait in line.

A different spin on fairness is to let people enter a lottery to let them buy what they want at lower prices. If you were to enter such a lottery and your number got drawn, you could then buy what you're after. Typically, submissions to this lottery are accepted during a specified time window. So if you have to be at work when the lottery first opens, that's not a problem. You have just as good of a chance as the first person who entered that lottery.

This was part of Saddleback Church's approach when they hosted the Saddleback Civil Forum with Barack Obama and John McCain during the presidential campaign. Since Saddleback Church is very high-profile, it was important that a purely pricing-based model was avoided. Taking an exclusively pricing-based allocation of tickets to see the candidates would have looked too much like favoritism towards the rich, a Biblical no-no.

Then again, Saddleback Church also had to cover its costs, so it actually opted for a brilliant hybrid approach that consisted of various tiers, each with a different price. It was a multi-tiered lottery. Higher-priced tiers would likely yield a smaller pool and a higher chance of being selected. You can't please everybody, but such a creative and enlightened approach likely pleased quite a few free market die-hards while at the same time placating those who were concerned about equality. If only governments and businesses were as wise.

Saturday, October 25, 2008

Look where you're baby stepping

When you're starting out from scratch, it's hard to predict what you're going to need. I just moved into a new apartment, and my kitchen cupboards are empty because I've decided that I'm not going to even try to predict what I might need in the future. The way I intend to re-acquire the supplies I'll need is by doing stuff.

I mean, sure, there's cooking oil, salt, and pepper. But for a little more exotic Chinese cooking, I'll need some sesame oil or star anise. Beyond the basics that I'm certain I'm going to actually use, it's impossible to predict what other odds and ends I will personally end up using. And soliciting advice from others in the past hasn't been good for me: what works for other people often does not work for me.

For me, the only way I have been able to know — with certainty — what I'll need is to actually start trying to do things. If I'm trying to figure out which ingredients I'll need to cook a meal, I just start trying to cook. If anything is missing, its absence becomes readily apparent.

These are kind of like kitchen use cases, if we're going to resort to software development parlance. It's like I'm going agile, or adopting Test Driven Development with my cooking processes. I could theoretically start out with nothing at all — to illustrate the method theoretically, I actually should start from nothing. I would soon find that I need a cutting board, a knife, and whatever ingredients my recipe calls for. For a basic meal I would also find that cooking oil, salt, and pepper come in handy.

After cooking a basic meal, I know with confidence that I have everything I need to cook — wait for it — a basic meal. That common use case is taken care of and I can cook basic meals in the future with confidence. Now, what if I were to go ethnic? I'd soon run into some cases where I'd need to stop and get some more ingredients.

The idea is that, after a while, I would eventually be able to confidently prepare a broad range of meals. This confidence is rooted in the knowledge that I have verified that I have everything needed — by actually having done it in the past.

Of course, it's possible to carry out this kind of method to absurd extremes. If you know for sure you're going to need it, then make a note and take care of it. Be reasonable in what decisions you leave for later and which ones you can make now. In the cooking example, it's great for didactic purposes to start from scratch, but in practice it's dumb not to have the basics like cooking oil or a cutting board.

What about when developing software? You can leave scalability concerns for later if you're just starting to write a web application, but no matter what, you are going to need a machine and a relational database. (If you're writing something, like an offload server, because it needs to be scalable, then you damn well better take scalability into account.) To maintain your sanity, it might even help to sketch out a data model. Take baby steps, but know where each step will end. Don't be content with merely knowing that your foot will be in the air and then end up somewhere on the ground at some point. After all, this nebulous "goal" can just as easily be achieved by tripping and falling.

Thursday, September 18, 2008

Getting more out of a sentence

There's a very simple, effective, and systematic way to amp up the amount of insight you get from the things that you read.

It's probably best to explain with an example. The following is from Ed Catmull's article for the Harvard Business Review, "How Pixar Fosters Collective Creativity."

We must constantly challenge all of our assumptions and search for the flaws that could destroy our culture.

How much thought can this spark in your mind?

If we just read this sentence several times, each time with an emphasis on only one word, we get a new angle on the general idea being conveyed. Something different is emphasized.

We must constantly challenge all of our assumptions and search for the flaws that could destroy our culture.

The emphasis is on the we. This tells me that whatever this sentence is talking about involves a team effort. Everyone on the team must be involved. It's a "we" rather than a "me." But does that necessarily have to be the case? It starts with somebody, right? Why not with individuals?

It's okay — a good sign, even — to ask questions about the truth or applicability of a passage. If you're a reader with a healthy sense of skepticism, those kinds of questions immediately arise. A lot of the same questions will come up: is it necessarily so? Is it true all the time? The more you ask these kinds of questions, the more you hash things out and figure them out for yourself.

Let's look at another word.

We must constantly challenge all of our assumptions and search for the flaws that could destroy our culture.

This time, the emphasis is on the word "search." This tells me that the flaws are often hidden; otherwise, they wouldn't require searching. Now, I'm an imaginative guy, so then I imagine myself searching frantically and then getting tired or discouraged. When I'm tired or discouraged, I ask questions. Why am I searching, anyway? What exactly am I searching for, again? How do I really know when I've found it?

That's only two words, and I could go on for a while, but I think the general method has been made clear. The biggest plus going for this method is that it is systematic; I can just emphasize each word until I'm plumb out of new thoughts. When I'm done with all the words in the sentence, I know I'm done.

Personally, I've found that this process also helps me internalize ideas. Even if I don't remember the exact wording of a sentence, the dance of thoughts anchored around a single idea tends to leave a lasting impression. I imagine this is because the thoughts that come to mind immediately tend to be those of the most concern to me. When it comes to things that truly concern me, I have an easy time remembering.

Wednesday, September 17, 2008

Ruby on Rails: optimizing a slow-rendering page

Recently at work, I was faced with the problem of fixing a slow-rendering page. The page is part of an internal content management system, where the content includes video, images, and text. We have over a thousand pieces of content being managed with this tool and the slow page was a listing of all the content along with key overview-type information for each content package.

Everyone was fine with this page loading slowly, since we knew that there was a lot of content. The problem arose when the page would only show around 50 items and then stop loading; the server configuration was set to automatically stop requests that took too long to process.

I knew that the best course of action would be to start measuring. I took a look at the logs to see how long the request was taking and the approximate tasks where time was being spent. At JibJab, we take pride in caching the hell out of everything. Only a tiny fraction of time was spent hitting memcached. And it looked as though memcached was doing its job: almost no time was spent hitting the database. But a ridiculous percentage of the time was spent "rendering" the page.

That meant the onus was on my code to speed things up. Since the slowdown had become more and more apparent as we added more content, my guess was that the loop was the main source of the problem. Anything that would be only a tiny bit slow on its own would lead to a delay over a thousand times greater.

The first thing I removed was a call to cycle(), which I was using to zebra-stripe the table display for easier reading. In its place, I put in an index variable that would be modulo-two to determine its index into an array of two choices (even row or odd row). To keep the same feature, I had to dirty up my code if I wanted it to run more quickly. And indeed that shaved off about one-fourth of the rendering time.

But the page was still taking over ten seconds to render, so I continued my hunt. A pattern I saw many times within each iteration of the loop was an old friend, link_to. I am not terribly attached to this particular friend, however, so I got rid of it and replaced it with some good old <a> tags, which bought me a good second or two.

So far, I felt pretty good about what I had been able to do to extract render-time savings. But there was still room to shave the time down, and luckily for me, Inflector.humanize was being called in each iteration like there was no tomorrow. I decided that there indeed would be no tomorrow for these expensive-looking helpers, and after removing all five calls inside the loop, I got total render time down to a very respectable two-second neighborhood.

The lesson learned? Sometimes code has to be a little messier so it can be fast enough to be used. Specifically, watch out for those beautiful-seeming Rails helper functions; they do a lot of work for you, but sometimes you're just better off doing the work yourself.

Tuesday, September 16, 2008

Bilateral versus multilateral trade agreements

News sources everywhere seem to lament the collapse of the Doha round of trade talks because countries would then have to resort to bilateral or regional trade agreements. For any supporter of free trade this would be understandable, but if we acknowledge the short term havoc wreaked by rapid changes to trade flows that do not give workers and business owners time to adjust, a gradual and stepwise approach to the opening up of markets seems to be a more prudent way to go.

One of the major problems cited by those who favor a broad trade policy is the inconvenience of having to keep track of different rules for different countries. I would go about solving this by building a hash table of trade rules. Basically it would mean that given a country and a product as input, we would get the policies regarding that product as output. This would be a large table, but we would ask and receive only what we are interested in.

There's also the advantage of isolated policies having limited effects, rather than opening up to the entire world in one go. Maybe there is something I'm overlooking, but I don't consider the use of bilateral agreements to be an indicator of failure. I see great potential for reaping the benefits of free trade with less of the shock typically expected by any opening up.

The collapse of grand schemes for international trade deals is really a familiar scenario that just happens to be playing out on a macro level: a project's planners have decided that they have bitten off more than they can chew, and decide to reduce the scope of the project. Everyone does this, and while we may feel disappointed at envisioning the results of the grand project and having to instead settle for something less, we eventually resign ourselves to the reality that we are doing all that we can, and it is a whole lot better than nothing at all.

Thursday, September 4, 2008

Colored bash prompt

The following will set a bash prompt to highlight the name of the directory in yellow, and in bold.

export PS1="[\u@\h \[\033[1;33m\]\w\[\033[0m\]]$ "
This will also work:
export PS1='[\u@\h \[\e[1;33m\]\w\[\e[0m\]]$ '
The escapes have to be in single quotes, however.

During a previous attempt, I wasn't terminating everything correctly, and so when my command exceeded the length of the row, it wouldn't wrap around and start a new line. It was not just a Mac OS X Terminal problem; I tried it in rxvt (installed via MacPorts) as well as the xterm that came standard with Leopard's X11.

Wednesday, September 3, 2008

Doctors and programmers

One evening, I was watching Mystery Diagnosis on the Discovery Health Channel with my brother. The show is about people who have strange ailments, but doctors can't figure them out.

It got me to thinking about how people in my profession approach problem solving. The approach that computer programmers take is up to us, and it's pretty open-ended, but it's widely agreed that we use measuring tools — debuggers and profilers — to examine whether each little step is doing what it's supposed to be doing. It's also a widely accepted quip that one should never try to guess where a slowdown or a bug is happening, because one is usually wrong about it.

But that sort of guessing without detailed measuring seemed to be exactly what the doctors were doing. For one guy, they drew blood sample after blood sample, but it turned out to be a problem on the genetic level.

I wonder if there's some level of professional ego that keeps doctors from doing what would otherwise be considered sensible. If it took a lot of lobbying to get them to use checklists, I wouldn't be surprised if they think they've gone through too much training to actually rely on detailed measurements.

To be fair, though, doctors probably don't have much scope to apply deductive reasoning on the entirety of the systems they're working on. There are no "development" or "testing" patients where it's okay to mess up and start over. Everything they work on is in live production mode.

Friday, August 29, 2008

Freeways that trick you

I don't like it when freeways trick me.

There's I-10 west into downtown LA, when it splits off to US-101 and I-5. Six lanes of I-10 turn into a single pathetic lane if you want to stay on it to keep going to Santa Monica. Most of the rest goes to US-101. Context over consistency, to serve the massive hordes going to Hollywood? Perhaps. But this is the mighty I-10 freeway, encompassing the width of the country and passing through Phoenix, San Antonio, Houston, New Orleans, and Jacksonville. It's not some little country road in the middle of nowhere. Someone should tell this freeway, "Hey, you're not done yet. You reached LA, but the real end is Santa Monica."

And then there's I-5 north to SR22 west in Santa Ana. When you're driving on I-5 and you choose the "Exit Only" lane to switch freeways, you should be rewarded for your dedication. But it is not like that at all. It really does mean "Exit Only" — and not to switch freeways. It takes you off the freeway and you have to switch into the other lane if you want to get on SR22 west to Long Beach. Well, you must be saying, it does say "Exit Only" so what were you expecting? Well, says I, it's clearly marked SR22 west to Long Beach, while it should really be marked as a regular exit, not a way to switch freeways.

Wednesday, August 27, 2008

Meetings are not always toxic

I'm a big fan of 37signals in general, but I remember a time when I went a little overboard with youthful zealotry for one of their philosophical tenets: meetings are toxic.

Back then, my business partner Steve and I had just landed our first major client, and this client asked for a kick-off meeting.

I curtly dismissed it, telling them I didn't need a meeting — leaving them baffled with my strange behavior (and Steve smacking his forehead).

Since then, I've found kick-off meetings with partner companies and clients to be crucial for greasing the wheels of personal exchange. It gets people talking to each other, which is worth pursuing because cross-company communication is a tricky thing. Plus, the stakes are higher: if you have to coordinate with another company, chances are that you're working on something pretty important. I've found in the vast majority of cases that it doesn't take extensive preparation to the standard I would expect of myself if I were to prepare for an intra-company meeting. People just like to talk and get a feel for each other. If meetings and conference calls are good for anything, they at least open up the communications channels.

After thinking about it for a bit, I realized that I initially learned to conduct business in a contrarian way, since I got most of my tips from the Web. But since then I've begun incorporating more traditional business practices because that's how most people still do business. I have a hunch that most people still do business "the old way" because it works.

Am I anxious that I could be blindly following new ideologies or anything that remotely seems to make sense, to my detriment? Would I have been better off doing business the traditional way and then finding out about these counter-cultural methods? I don't think so. Whichever end of the spectrum I may have started on, the important thing is to continually evaluate the merits of each idea I encounter, whether old or new, and knowing why I do things the way I do. The truth lies somewhere in the middle, and it's reachable starting from either side.

Sunday, August 24, 2008

The introverted leader

During the course of my two years attending Discovery Christian Church in Davis, I had the chance on several occasions to eat lunch with the pastors, namely Aaron Brockett and John Richert.

I still remember the most surprising thing that Aaron ever mentioned to me. He said that a lot of the leaders at Discovery, himself and John included, were natural introverts. From the outside, it seemed that they were anything but.

They greeted people left and right, confidently gave their sermons, and coordinated the big picture very smoothly. One of the things that caught my attention at the beginning, and subsequently hooked me in to attending regularly, was how the services on Sunday were so well-run.

As for me, I have found that I am quite happy to be a follower, unless there is a leadership vacuum — either no leadership at all, or very poor leadership. To be fair, I've found that very poor leadership can come from both extroverts and introverts.

In extroverts, the leadership pitfalls have to do with the tendency to make decisions without thinking very hard about them; not being aware of how everyone else feels; or not stopping to ask what everyone else thinks.

With introverts, what I've observed includes the lack of dynamism to keep everyone interested; indecisiveness while trying too hard or too long to build a consensus; and a lack of forcefulness.

Being the introvert that I am, I naturally got to thinking. What, I asked myself, can I do to leverage my natural strengths while avoiding the pitfalls of my natural weaknesses?

My natural strengths are things which come easily to me: thoroughness in exploring issues; attention to detail and a concern for total correctness; and having a passion for learning new things on my own, which makes it easy to become knowledgeable in various areas. I see these kinds of traits as "hard-wired" where my thirst for knowledge or obsession with detail give rise to good things. Like a microprocessor is hard-wired to perform basic operations like addition and subtraction, I'm hard-wired to do certain things because I'm comfortable with them and I know how to do them.

But the desirable traits that extroverts possess that I don't have — outspokenness, dynamism, gregariousness, forcefulness — I have to run them in emulation mode. I have to figure out some software to run on what's hard-wired in me so that I can do pretty much the same thing. Software operations are slower and require more instructions than hardware-level instructions. If we're going to continue carrying our analogy over to people, this means that I've got to put more effort into expressing qualities that are generally characteristic of natural extroverts.

But it's possible to do so, and this is good news.

For the introvert, it just takes a lot of piecing together. For example, the extrovert can just "be forceful," while the introvert has several mental steps to take. First, he must recognize that forcefulness will be necessary to avoid wasting everyone's time, and it's better for everyone. Then, he must double-check that the position he is strongly advocating is correct, because he knows his confidence in a position relies on knowing that it is as correct as can be, given all the data. Then there's recognizing that the best that anybody can do is make a decision based on all available information, and that's really the most that we can do. There is no sense in second-guessing since more information is not available. Finally, there's actually saying what needs to be said, or acting out what needs to be done. This requires a conscious mental step for the natural introvert.

As anyone can see, it can be much more involved for an introvert to display extroverted qualities, but it is possible. Over time, the basic behaviors can be optimized and moved a little closer to being hard-wired, just as software can be extracted into microcode to sit somewhere between software and hardware. This way it runs more quickly and more readily, with less effort.

There's one last thing before I completely beat the dead horse of this analogy: even though things done in hardware run much more quickly than those which are run at the software level, the software level allows for much more richness and flexibility. What does this mean for natural introverts who are looking to be effective leaders?

My very introverted answer is that I don't know; I'm still trying to find out.

Friday, August 8, 2008

Where software developers fear to tread, or why open source fails

Open source has had its triumphs and failures.

The most notable wins include Eclipse, Linux, Apache HTTP server, MySQL, and Mozilla Firefox. Developer tools and server software are generally areas of strength for open source. I wouldn't say that Mozilla Firefox is an anomaly. Rather, it shares something in common with developer tools and server software: the programmers who work on it are also its users.

Unfortunately, this is not the case with the gaps that open source has so far failed to close. A friend of mine recently asked me about my choice of spreadsheet software. We talked about it for a bit, and agreed that OpenOffice.org Calc is still clunky. I suggested Gnumeric, but I don't really hear anything notable going on with it these days. In short, I don't see an Excel killer in either of these. It's not for lack of vision that these spreadsheet projects have stagnated. It's for lack of sustained passion and developer interest in making these products world-class. Rather than get deep into using these spreadsheet packages, I often just resort to SQL queries and image generation libraries if I need a chart. And I'm guessing that other developers have also resorted to what was more familiar and efficient, given their esoteric knowledge of more powerful tools.

It's obvious to many of us that successful open source projects are characterized by programmers who are also users, but I would go a step further and say that the programmers must not merely be users; they must be the power users. There are two ways that a programmer could be a power user of his own software: out of enjoyment or out of necessity.

Open source software developers should be the power users of their own software. This sustains the passion to "scratch the itch" and fix bugs to produce polish. With proprietary software, programmers have the benefit of separate QA teams or product managers. In open source, it's overwhelmingly the case that they do not.

It's been said that with enough eyes, all bugs are shallow, but it's quite often the case that not all of the mouths will bother to report what the eyes are seeing.

Monday, August 4, 2008

The music at 24 Hour Fitness

For my morning workout, I switch between two different 24 Hour Fitness locations. One is right by where I live, in Irvine, California, where people tend to be either white or some kind of Asian. The other gym, near Long Beach in the city of Carson, is on my way to work. Most of the customers I see there are either black or Filipino.

Each location plays different kinds of music.

At my neighborhood 24 Hour Fitness, the music that's played is the angst-ridden punk and alternative that people have come to expect out of Orange County. (To be fair, I like a lot of the songs they play.)

In Carson, it's a bigger gym with some sort of Magic Johnson approval. They make a big deal of being tied in with Magic Johnson, and his pictures are plastered over all four of the walls, on both floors. If I were Magic Johnson, I would be a little embarrassed to have so many pictures of me smiling statically at people as they work out. The music is hip hop and R&B. As both workout and leisure music, it works for me. (In fact, it recently helped me rediscover the genius of Usher.)

Noticing the difference, I got to wondering who makes the music selections. If it's an official policy that the music fit the ethnic demographics of the community where the physical gym is located, then I'm pleasantly surprised that they took a factor like that into consideration.

Saturday, August 2, 2008

Mexican flower vendors

One my drive home from work each day, I see Mexican guys trying to sell flowers, standing on the curbs by freeway entrances and exits that have stoplights. After seeing them all week, I got to wondering whether any of them ever make a sale: I haven't seen anyone buy flowers yet. And I'm sure they make a sale here and there; otherwise they wouldn't come back.

That got me to wondering whether there's some sort of demographic they target.

Given that the particular time of day is when most people get off work and then sit in traffic, the cynic in me thinks that the buyers are probably making their purchases for reasons other than being considerate. A man doesn't have to plan the purchase of these flowers in advance, and even if he did, it's not guaranteed that the seller will be there.

My hypothesis is that these traffic light flower vendors' target market is men whose wives or girlfriends are upset that they're coming home late from work, and these busy guys are grasping for any quick fix. In that kind of situation, there's a lot of potential to jack the price way up. Personally, I am not yet decided on whether I would ever buy flowers from a Mexican traffic light flower vendor, but I wonder if, next time, I should at least ask them for a price in order to get more information.

Sunday, July 27, 2008

Foreign key cascade on update dangerous?

In relational databases there's a feature called foreign key cascading. Typically, there's an ID associated with each row, or entry, in a database table. Tables may refer to other tables using a foreign key, which typically refers to the IDs of the rows being linked. There are options to "cascade" on two kinds of operations: delete and update. When a table refers to another table, the table pointing to the other table can be configured to take changes in the ID of the original table and integrate them automatically. This is what happens when ON UPDATE CASCADE is set on a foreign key. When ON DELETE CASCADE is set on a foreign key, deleting rows in the original table will also delete rows that are pointing to that row.

So, it's not hard to imagine situations where cascading on delete would be dangerous.

Now a while ago, I was talking to a database specialist about cascading updates on foreign keys. He mentioned that they're dangerous, so I took his word for it. But I've been thinking about it and I still don't see any way they can be dangerous; the only changes being propagated outward will be changes in the values stored in foreign keys; no rows will be deleted.

I've asked around to get second opinions and examples of situations where this would be bad; for now, I'm just going to avoid cascading updates for future projects.

Sunday, July 20, 2008

Truth never changes — or, why we're solving the same problems

"Truth does not change. What was true 2,000 years ago is true today, and will be true 2,000 years from now. New truths may be discovered, but that just means that it was true all along."

The above is paraphrased from a sermon given by Pastor Rick Warren at Saddleback Church. Very few things are really new, which is consistent with my observation of what's going on in the world, and if they are, chances are that there's some catch. What's new and doesn't last probably doesn't work or just isn't true. Such things lack conceptual integrity and fall apart eventually when reality hits.

(I've just started a new job at JibJab, on which I'll write more about later on, but that doesn't mean my mind is completely devoid of all thought dedicated to solving larger problems.)

When I see some new file storage startup, or some new effort to create an operating system, or several companies trying to push their respective next-generation platforms, I think, "Oh, great. Not another file storage startup. Not another operating system. Not another platform."

But I see these things happening again and again — first in Web 1.0, then Web 2.0 — because the problems haven't been solved. The truth in all of these situations is that the problems remain, or have resurfaced due to technological change or heightened consumer expectations.

Having a lot of competitors trying to tackle the same problem shouldn't be seen as, "Hey, there are just a bunch of me-too companies trying to tackle old problems." Instead, it should be seen as, "These are problems that continue to badger us. These are important problems."

The people starting these companies see the importance of addressing these "old" problems, but it's important that rank-and-file employees of these companies, as well as consumers and journalists, recognize the significance of the larger mission.

In order to appreciate these efforts for what they are, it helps to focus on the fact that solving the problem is a formidable task — rather than focusing on the fact that the company is merely another contender in a crowded field.

Monday, June 23, 2008

Proofs by induction and real-world business needs

While I was a young and eager undergraduate computer science student, I was taught the method of proving things by induction. Basically, it's a way of showing that something is true for some simple cases, and then showing how it can be true for more complicated cases. There are two major parts to it: prove that the base case works, and prove that the inductive case works.

I see some applications of that kind of thinking in everyday business. Just because I can manage documents on my own doesn't mean that the way I do things will scale to other people. So, to build a solution that grows with the business, I'd have to show first that it works for me, and then assuming it works for X people, show that it will still work when X+1 people are in the picture.

The thing with most inductive proofs is that they show that something is true, or if we map it tightly, it would show that things are possible in business -- that something is computable and knowable. But in real-world concerns, efficiency is a big factor: while it's necessary to show that some method or process works, it is not sufficient. We've got to show that it's efficient, too. (Or in other words, "Duh! We know it's possible. I'm concerned with how quickly and cheaply can we do it.") So, I'd have to show that something works and is efficient for me, and then that if it works and it's efficient for X number of people involved, it will still be efficient for X+1 people.

Additionally, the method or process that we try to prove by induction would probably look like a step function to handle different domains. The reason for this is that a group of five people is going to present a different kind of situation than a group of 100 people, which will in turn be much different from managing an organization of 10,000.

Note to self: when trying to come up with a new solution, come up with something that looks like a step function, with different methods for different points in an input domain, and prove that it works and that it's efficient.

Monday, May 26, 2008

Put Apache Tomcat in its own directory, and your web applications in another

I recently took my first look at Apache Tomcat 6.0. I was inspired by Tomcat: The Definitive Guide, 2nd Edition, which covers Tomcat 6.0. It mentioned a way to separate the actual Tomcat distribution and binaries from the webapp data, but described it for environments that had it pre-installed as an RPM or DEB package.

As I'm writing this, the latest version is 6.0.16. Here's what I had to do to get my webapp code in a separate directory tree from the Tomcat binaries as downloaded straight from tomcat.apache.org.

  • Create a directory, say /home/joshuago/domain.com, where all the webapps for that domain are going to be.

  • Set CATALINA_BASE to the directory above. (Don't set CATALINA_HOME.)

  • Move conf, logs, temp, and webapps directories to /home/joshuago/domain.com/.

  • Run apache-tomcat-6.0.16/bin/startup.sh script as usual.

  • Start developing at /home/joshuago/domain.com/webapps/ROOT/.

Change the directories as convenient or as suits best practices for your particular environment.

If you have multiple sites (and multiple instances of Tomcat) you'll need to edit conf/server.xml to make them listen on separate ports.

Thursday, May 15, 2008

A fix for LaTeX "Missing $ inserted." console message

While I was running pdflatex on a .tex file containing a math expression, I got this error message:

Missing $ inserted.

Powering through it would evaluate the math expression adequately, but I included a plus sign later in the same sentence. I followed the directions and made it convert to a PDF beautifully by adding a dollar sign at the beginning and at the end of the math expression. (It's not just at the end-of-the-line, like I would normally guess it meant in a Unix environment.)

So, what started off as 9*2^4 turned into $9*2^4$ and produced a beautifully formatted PDF.

I should also note that Norm Matloff has a great introduction to LaTeX.

EDIT: This one's even better. It just took a while to load and I got impatient.

Thursday, May 8, 2008

Observations about making money from open source

Open source is a great thing and I've personally benefited tremendously from it. It has let me deepen my knowledge, given me entire software platforms to invest my energies in, and has opened up a wide landscape of job opportunities. For a company though, it's pretty challenging to make money from it, and I recently I got around to thinking what successful companies in open source have done.

Arguably, Red Hat is the most successful open source software company out there. They came to mind not only because their product portfolio is mostly based on open source or that they've been around for a long time (relatively speaking), but because I actually use their products and find value that I willingly pay for. Good for them. But I wanted to figure out what it was they did in very general terms, because I'm definitely not going to try to take them head on.

Open source software has its strengths and weaknesses, where the strengths and the value produced often tend to accrue to the users and programmers of open source software, and the weaknesses plague them. Switch this around to the point of view of a company and you can actually get the opposite scenario: the strengths of open source software make it difficult for traditional software companies to make money, while the weaknesses in open source present an opportunity for an enlightened or creative software company to step in and fill the value void.

The strength in open source software is drawn from its unrestricted use and the consequences of unrestricted use. Anybody can just take the software, make changes to it, and repackage it. There are no restrictions on the use and distribution of the software. And since anybody has access to change the software, they often do — and the software often experiences a lot of change in a short period of time. In short, open source software is pretty great if you're a user: you can take it and do with it what you like, and there are always new features coming out so you never get bored or have to be stuck with the same old thing for long.

The weaknesses of open source software are analogous to its strengths. You could say that it's just picking up the other side of the same stick. Sure, open source software is easy to change and great for sharing, but when everyone's allowed to do this, there's a strong tendency towards chaos. Some efforts, like the Linux kernel or Mozilla, have very strong centralized cores that have survived challenges, but many others like XFree86/XOrg and FreeBSD/DragonFlyBSD have had very public internal debates. The fast pace of development also has its downside: frequent updates to existing software come out, but they come at such a quick pace and have not been well-tested. It's a hit or miss proposition when upgrading any reasonably important server to the latest version of a software package: it could break other software packages that haven't kept up. How does one get access to bug fixes or security patches and be assured that there are no unintended side effects?

Based on these observations and what Red Hat (and others in the industry) have done, here are a few guidelines on trying to make money from open source software.

Don't try to sell value that you didn't create

It's not a moral recommendation but a practical one: even if you tried to sell a repackaged version of what other people produced, nobody would buy it, because it's available elsewhere for free. You shouldn't just repackage the software and sell it like you would a traditional software package. Technically, you could repackage it for convenience, but you won't be able to make much money by charging high prices for your software. So there goes that plan.

Don't expect to sell value that someone else in the community can create/duplicate

Now, it's fine if you want to make enhancements to a piece of open source software to contribute something back to the community. Many companies do this: IBM, Red Hat, Novell, and Sun, just to name a few. Users really win big with open source software, because any feature in it will be available for free. Unfortunately, if you as a company try to tack something on to the project and sell it, your competitive advantage quickly erodes because you have to open up the source code (at least when using the GNU General Public License).

I say don't expect to sell value here because you can certainly try, and in some cases, succeed. But I'd say don't count on it. If you do want to try, make sure you enhance software that's covered by the BSD license, not the GNU GPL. If you choose the latter, you'll certainly have to open up the source code to any modifications you make. With BSD licensed software, you can keep it closed but include a note of acknowledgment in your software's licensing information.

Focus on what open source software finds difficult or impossible to address

With that said, open source software still has its weak points. All the openness can and does lead to fragmentation and inconsistency. Because people working on a project work only on what they want to rather than have a boss dictate items to them, a lot open source software lacks that extra polish that you'd expect from a consumer- or business-ready product.

I have a hunch that this is why open source programming tools are so clever and refined, while open source products meant for consumers seems to have a hard time getting past alpha or beta quality. Pretty much any open source program that a programmer would use on a regular basis and use heavily is bound to be very well done. Any rough edges are quickly smoothed out. But when a tool is built for non-programmers, the overwhelming tendency is for it to languish in mediocrity and stay under the radar of the wider community.

The business opportunities in open source software are rooted in the fact that open source developers overwhelmingly favor working on things that they want to work on. This means that the parts that aren't so fun are left undone. There is this desert where a programmer's passion only occasionally meanders — but it is wide open for businesses to step in, where another incentive which many of us are very familiar with, money, can take hold and flourish like native desert flora.

Where grassroots open source development stops, profitable enterprise begins. Here are some ways to make money from open source.

Client interactions and case-specific integration/support. It's fun to write new software, but not so fun to make sure that it works for every single person out there. Fortunately for the businessperson with an interest in open source, there are people who are willing to pay in order to get open source software working for them. Someone once said that open source software is only free if your time is worth nothing. There's truth in this, but that time can be dramatically reduced while avoiding being locked into closed source, proprietary software. Red Hat, Novell, IBM, and Sun make some decent cash through support contracts and servicing.

Subscriptions to updates that are well-tested and certified. There are software updates, and there are software updates that are guaranteed to work. I think Red Hat was extremely clever to set up their subscription system, and I admire it because I know that the updates coming from it aren't going to break anything. They do the hard work and make 95% margins on subscriptions. I can get on with my day rather than worrying about properly upgrading my server. It's unquestionably a win-win situation.

Coherent visual/brand identity that promises a certain expectation of quality. The open source community is vibrant, but it's also prone to fragmentation. Moreover, the traditional business practice of building a sustained competitive advantage through branding still stands in this new world of open source software. A brand still holds sway, and still serves as a focal point for trust. Novell, Red Hat, and Mozilla Firefox are highly respected open source flag bearers. It's really good once you get to that point, but, as in times past, building a brand requires a lot of hard work over many years.

Provide a stable platform: still open, but rarely changed because it has to work. There are two big problems with open source platforms: fragmentation and rapid change. Red Hat and Novell provide Linux distributions that can serve as stable platforms on which to build. They're infrequently updated and tightly specified. Ubuntu also has a Long Term Support (LTS) version of its releases, meant for the same purpose. It can be very rewarding to a company when it has control of a popular platform that holds a lot of developer mindshare. The vast majority of developers just want whatever they're building on to be predictable and for people to use their software. Providing a stable platform, trusted by developers and freely available to users, is a great way to establish thought leadership and make a name for your business. It yields a lot of advantages if you can get to the point where you are the trusted caretaker of a platform, but like branding, it's hard work and it's a long process only for the very committed.

Saturday, May 3, 2008

Why the office of U.S. President should be available to all citizens

Under our current Constitution, an American citizen is an American citizen — except when it comes to the presidency.

There are American citizens who will never be able to attain the highest office in the land, even with distinguished military service, community involvement, or a track record of political achievements. The one differentiating factor that isolates these citizens from the rest is that they happened to be born in another country, to parents who were not American citizens. This means that foreign-born American citizens — whether they were born in Mexico, the United Kingdom, Canada, Germany, China, or India — can never be President.

Nevertheless, the United States is one of the most accepting countries in the world when it comes to citizenship. Acquiring United States citizenship is a long and arduous process, but in theory it is open to all. And as many of us know, there's so much to be done and so much that could be done aside from coveting the presidency. In the political sphere, state governorships, seats in the Senate, and Supreme Court appointments are technically open to naturalized citizens. In the economic sphere, immigrants are increasingly providing the needed brainpower to develop new industries: a quarter of Silicon Valley startups were founded by entrepreneurs with a Chinese or Indian ethnic background.

Things are pretty great for the naturalized American citizen — but they could be better. There are a lot of good reasons to amend the U.S. Constitution to get rid of this last remnant of inequality, and not a single good reason to keep it as is.

Reasons to Amend

What we have now essentially delineates Americans into first- and second-class citizens. Granted, fundamental rights have been addressed thanks to the tireless work of the many who have come before us. And it's true that holding public office as a representative of the people should be viewed more as a privilege than as a right. But to lock people out of the presidency in the Constitution, for circumstances that were completely outside their control, creates an undeniable division of citizens into first- and second-class citizens. That's just the simple fact of the matter. Certain citizens get one more privilege than the others do. This distinction does not mean much to most citizens and does not impact them directly — running for President is not going to be a practical reality for the vast majority of Americans — but to be able to say, with absolute veracity, that our Constitution creates a system of first- and second-class citizens is just downright embarrassing when we claim to be the purveyors of equality and the beacon of democracy to the whole world.

What we have now does not effectively address the original concern with illicit foreign influence, and it is a blunt instrument in attempting to address this concern. The requirement for the President to be a native-born citizen was included in the Constitution during a fragile time for America, where the danger of foreign influence was very real, and whose potential consequences could have destroyed America early on. In short, it was justifiably paranoid and the drafters of the Constitution were right to include it at that time. But we have no justification for keeping it now. Voter sentiment about foreign influence and mixed allegiances presents the biggest barrier to ratification, and I'll expand on this point further in the next section. But in short, the current clause does not serve its purpose — ensuring loyalty and allegiance to the United States of America — and there are better ways of demonstrating loyalty and commitment.

Ratifying such an amendment would do much to integrate immigrants into the fabric of American society. By sending the message that immigrants, once naturalized, are truly equal in the eyes of the law by being eligible for the highest office in the land, this amendment would give them a stake in America's future. It could also lead to a representative figure.

Further Elaborations on Foreign Influence

It's completely valid to ask for a sign or indicator of loyalty to the United States. Where you're born is a blunt and ineffective instrument for doing so. If anything, let one's military service record be the testament of loyalty and dedication, not one's place of birth.

The original intention and the situations surrounding the initial amendment are not valid now. Locking a restriction into the U.S. Constitution was a necessary precaution to very real concerns. Keeping it locked in shows a mistrust in the American people to select a good Presidential candidate.

Accentuating the Positive

For those who are overly pessimistic about the current situation, let me just give an overview and explain why I think the current system is actually quite good. By accentuating the positives, my intention is to frame the debate in more forward-looking viewpoint that's conducive to progress, and not wallow in despair over the hopelessness or xenophobia inherent in the situation.

The current system does allow for children of immigrants to become President. If you're born in the United States, you are automatically a natural-born citizen, even if you are a Gonzales, Zhang, or Singh. That part is taken for granted, and in arguing for something better I certainly don't want to take the current system for granted. Rather, I want to make it a point to celebrate the tremendous opportunities already available.

Closing Remarks

There's some merit in the argument that political energy should be diverted to more pressing concerns like the economy, health care, transportation. There are more important things to address in the meantime.

Some would argue, "If it ain't broke, don't fix it." But I'm saying that it's broken. Leaving rotten parts in the system leads to decay. But I acknowledge that, however simple it is, it's still a small issue compared with the bigger problems facing us today.

Since this is such a no-brainer, then, I suggest that it be put forward for passing as a bipartisan measure to encourage working together across the aisle, especially when more divisive issues take center stage. It will be a symbolic gesture to the American people and good practice for our elected representatives. I would welcome a well-argued case for keeping things the way they are.

A lot of times, the status quo is there for complicated reasons that are unknown to us, and set up in a way that we should not try to tamper with. I do not believe that this is one of those cases. The issue is simple, and all that's holding us back is our own reluctance and inherent mistrust of change, even if it's clearly for the better.


Help not wanted (The Economist)

Should the Constitution be amended for Arnold? (USA Today)

Friday, April 25, 2008

A pleasant surprise from Pentel

I was raised on Quicker Clicker mechanical pencils from Pentel. Back when I started using them as a wee lad, they felt heavier and sturdier since they were crafted out of more metal content. Now, Quicker Clickers are mostly made of plastic. But like all Pentel products, they still carry a lifetime warranty.

For pretty much anything else, I don't care about lifetime warranties. First of all, I'm not going to take the trouble to dig up my receipt for something I bought years ago. Second, I just don't care that much about most of the stuff I buy.

But I take my writing implements very seriously. For mechanical pencils, I prefer the Quicker Clicker over anything else. For erasers, I like the Staedtler Mars Plastic but I'll settle for a ClicEraser from Pentel. For ink pens when writing things in my planner, I will go out of my way to get a Uniball Signo RT Gel, 0.38mm in either black, blue, or red.

I keep several Quicker Clickers around at all times to ensure optimal performance, but recently one of them broke. I've sent broken pencils to Pentel before, and they've sent back new ones — without requiring the hassle of presenting the original receipt. I knew that they followed up on their lifetime guarantees.

But this time around, they not only sent it back new, but also filled it up with lead refills and included a separate 12-pack of Pentel lead refills.

Quicker Clicker plus Lead Refills

It's a small gesture, but it's much appreciated from this customer. Thanks, Pentel. Your products continue to be expensive up front but they are totally worth the premium price.

Monday, April 14, 2008

Growing responsibility on a solid base

People who cannot handle small, easy duties should not be entrusted with more power and responsibility. Basically, be faithful with the small things, and you will be entrusted with bigger and better things.

I particularly like this lesson because I love the process of building and focusing on smaller tasks, but at the same time I've always been a big dreamer. As one would expect, though, I also experience that impatience to move on to the next step, even if I'm nowhere near ready.

But that's the way the world works.

I've been thinking too, that the ability to wield power and responsibility is not entirely intrinsic. True, you can be put in charge of small things as a test of your potential, which is a property that never changes, and it does serve as a way to winnow out the good from the bad.

But I realized that the process and experience of handling the little issues and tiny problems is itself critical to bringing out any latent potential or untapped talent in people. So essentially, you've got to be trustworthy in the first place, but handling the little things is crucial practice for the big leagues.

Monday, March 3, 2008

Reading old translated classics

The past couple of years, I've tried to become a more cultivated person by trying to read the "classics" — old books revered by well-read people far and wide. A lot of literary contributions have been in languages other than English — the French and the Russians in particular have made tremendous contributions to world culture — and in reading some French and Russian classics I've wondered how much I'm missing by not reading them in their original languages.

What I've found has been utter genius as far as pure ideas go, and eloquent choices in figurative language. Both translate well, as one would expect. As far as clever turns of phrase, I know I'm missing out because the wording hasn't seemed terribly impressive in the translations of Les Miserables (by Victor Hugo, originally in French) and War and Peace (by Leo Tolstoy, originally in Russian with a lot of French phrases). Even in translation, though, I found Victor Hugo to be frequently poetic even in prose because of he often pulled in some gorgeous metaphors.

Tolstoy was the same too, only his were more gritty. Sure, he wrote grandly of the ocean of humanity, but he also compared the movements of armies to the inner workings of a clock, made some references to ideas from calculus, and compared the actions of humanity to the steam engine.

Reading long 19th century novels in translation, I admit, was an effective way for me to relax and fall asleep at night. I pressed on because the stories were interesting enough. Still, though, with the attention span of most people these days, I think I know why the new books released tend to be much smaller than many of these "classics" that I read. Either readers are more impatient and pressed for time these days, or authors have become better at getting their points across with fewer words.

In any case, it'll be good to mix things up by reading something shorter and something that was originally written in English, so I've started reading All the King's Men, a much more recent American classic.

Friday, February 29, 2008

Project idea: structured financial data

As they are, things are pretty good for everyday investors on Main Street. The United States Securities and Exchange Commission makes the filings of public companies readily available on its website. If someone were interested in researching a company thoroughly, pretty much all the information is there: how much debt a company is carrying around, what kind of revenues it's pulling in, what the company's assets are worth, and so on.

So, things are pretty good, but they could be a lot better.

Now, I say "investors" and not "traders" because traders will typically need more than just information from the SEC. It's also a clear distinction that will limit the scope of the project I'm about to propose so that it's very clear what will be needed (and, more importantly, what will not).

The current situation

All this data is in HTML. It's great for human readability. Right now, that's exactly what everyone's doing: reading them one by one and poring over them. This is probably because there are a lot of loose, interesting facts that are specific to each company's way of reporting things. But among all that loose, unstructured data are structured tables containing information like a company's balance sheet, its cash flow, and its earnings.

For a stock market related project my team and I are doing, such data in a structured format would go a long way. Currently, there are data providers who would provide this kind of thing for a monstrously large monthly fee, perhaps because they bundle it up with other things. But this is public data. Nobody should have to pay a fee for it.

I personally tried to write a parser to automatically pull this data. I was reduced to entering some data by hand, but with around 8,000 companies to deal with, that would take me a couple of years, full-time. There is no way that this is feasible. Entering data by hand was pretty easy, but really tedious. What a human being can deduce by a quick glance, an automatically programmed parser would have trouble with.

Such a task would be, in my opinion, well-suited for being built and maintained by an open community. It's a big job — not particularly hard, just a lot of work. It doesn't make sense for any one company to feed these in; the ones that have this data are charging high rates for it. Everyone should be able to benefit from it, so contributors should include people from various organizations.

A proposed solution

The best solution I can think of would be to take the lead on the project initially. There would be a project website running on a simple web application. Since the number of companies is known, there would be a frequently updated status report saying what percentage of the task is done. For example, the front page could say "3000 out of 7500 companies with complete data" or "400 companies with incomplete data." To ensure the project's open nature, there would be a daily or weekly dump of the database. That way, no one party will control the data completely. Additionally, the community would have a reasonably complete backup.

It is essentially a massive data entry project. Not so fun, but for the participants there is potential fame and fortune as well as a strong mission. Once the data is available, programmers and financial analysts can graph data that would be buried in tables and easily compare data for multiple companies in a timespan on the order of a few seconds, rather than the current process, which could take minutes, hours, or even days.

The fields that would be stored would be standard items on a company's balance sheet, earnings statement, and cash flow statement. Eventually, if further patterns can be found, more fields would be added. There is a much more ambitious initiative which attempts to address the problem of all the fields, which I will discuss below.


Right now, there's a major push underway to address this exact problem: XBRL, which stands for eXtensible Business Reporting Language. I looked into it several times, and have found that it is characterized by slow progress and a quiet reluctance from a lot of the parties involved. Companies in the XBRL test program are few and far between. A lot of big names in financial research are entrenched, and they depend on keeping this structured data as revenue sources. It should be noted that if XBRL somehow succeeds, this open community project I'm proposing would not be needed at all.

The eventual XBRL specification that would be used for public companies' SEC filings would have structured fields to address all the data reported, but this could be a potentially huge problem because of the learning curve for new software tools that report preparers will have to get used to.

If there are any interested readers out there, please leave a comment and let me know what you think. This idea has been in my head for a couple of weeks now and I'm certain I left some gaping holes in my discussion of what I have in mind for this project.

Thursday, February 21, 2008

The gospel for stock market junkies

One of the recurring struggles among people who are part of the evangelical Christian tradition is finding the balance between faith and good works — knowing where each of them fits in and what they're for. Since I've been trying to read three chapters of the Bible every morning to get my bearings before jumping into what are typically full days, I read over something that reminded me of this.

Now, the "full days" have been dedicated to learning about stock market investing and exploring various angles for the individual investor to take when selecting stocks.

The price of a company's stock takes thousands of factors into account. One way to split up these factors is to consider 1) the book value or intrinsic worth of the company, and 2) the other things like projected growth, macro-economic concerns, takeover speculation, and a host of other factors that are not directly tied in to the numbers that the company is posting.

The stocks of most healthy companies will trade at prices above their book value. But when they're hammered by a bad economy or growth slows down, they won't go down to zero if they're solid companies, because people called value investors will swoop in and "support" the stock at a price at or somewhere near its book value. It's like a safety net, and these value investors keep stock prices at certain levels so they don't go down too far.

In good times and when a company's prospects look good, its stock will trade above the price set just by the raw numbers, whether the numbers are already in the bag or just strongly projected. At every point though, it's supported by its raw value, and every other gain in price is tacked on as extra.

I see us like stocks, and our spiritual well-being like the price of a stock. What God does for us is He gives us that solid book value. To get our balance sheets and cash flow in decent working order, so to speak, we must have faith and accept the Lord's gift of forgiveness. Then we're fixed, and our intrinsic worth is affirmed; there's no taking that away. We may be battered by the storms of life but we don't sink down past a certain level, because God sees value in us.

That faith can be enough to get by, but just as investors want to see a stock take off, I think it's in our interest also to grow, and not just rest on our laurels like some stocks languish at seemingly constant lows.

Faith gives us something like the value investor-supported base and ensures we don't sink past a certain low, but doing good deeds gives us growth and dynamism. On top of that, our good deeds often feed back to strengthen our faith, just like a company making constant process improvements and aggressively pursuing business will see the results feed back into the bottom line and strengthen the base of its value.

Tuesday, February 19, 2008

Freedom is shabby and inconvenient

As much as we value liberty and tout the virtues of living in a free society, I humbly submit that eternal vigilance is not the only price we pay for our freedom.

Anywhere there's freedom, you will find disorder, inconvenience, and shabbiness. We allow our freedoms to be restricted because we want some sort of orderliness, convenience, or good aesthetics. These are good things, but conceding to rigidity often means losing substance or meaning.

Take homeowners' associations, for example. With pretty much any HOA, you cannot get away with painting your house bright pink, letting the weeds grow to four feet in your front yard, and then parking your car among the weeds. It may be how you prefer to live your life and run your household, but by joining the HOA you are giving up freedoms that you would otherwise have, in exchange for the assurance that your neighbor across the street will not paint his house a loud pastel purple, because pink is okay but purple is not. You may be fine with how shabby your house looks, but you'd rather your neighbor not make his house an eyesore for you. So what you end up with is people being restricted; everyone's content, but chances are that nobody's house really looks any different from anyone else's.

This year, I'm living in Irvine, California. Having lived in more freewheeling places in California, like Davis, or San Bernardino, or Monterey Park, I feel the contrast here. Everything is much more convenient and well-planned, but it's hard to find shops that aren't part of a large national chain. In place of character and charm, I've got well-run homogeneity — which I actually like, but which plenty of others detest. (Even San Bernardino, with neither charm, character, nor well-run homogeneity, had good Mexican food.)

It's kind of like this with people, too. People may dress sharp, but I've found that many people who have taken the time to develop character on a deeper level, or hone their talents, are not the most fashionable or up to date when it comes to the latest trends. Of course, if you were really smart and considerate, you wouldn't be a complete slob and smell utterly foul due to all the time you spent "finding yourself" with little thought of more trivial things. All I'm saying is you just have to look good enough.

Or take a look at any Linux distribution, and compare it to Apple's Mac OS X or even to Microsoft Windows. Running Linux is the ultimate celebration of freedom, if you are using a computer. But you have to go through a lot of trouble to get wireless networking going, or even printing. The font display on any Linux distribution is mediocre at best, but you have access to change all of it, if you are so inclined. The trouble is, it's not fun at all to make these changes, and a lot of these difficulties arise due to inconsistencies and disagreements among all the "free" people about how to do things. With a proprietary, tightly controlled platform like Apple's or Microsoft's, there is much less freedom to tinker, and you have to trust that they make things "just work" but that's precisely why one would give up one's freedom. Most people just want to make things work.

Sometimes, we give up absolute freedom because it's more convenient to do so. Those who restrict us should give us something good in return, but I hope we don't lock ourselves in so tightly that there's no going back. Liberty has a price, and on top of that, it's not always pretty.