Havoc Pennington

Spring is here

Sun, 22 Mar 2009 20:45 -0500

Haven't had time to blog in months ...

Have been working on litl during nearly all available hours... it took some time (doesn't it always?) but we have a great team assembled and it's a ton of fun to focus on writing the code rather than hiring, and see the rapid progress a complete team is able to make. (If you want to pile on, we aren't urgently hiring, but we probably will be at some point especially for the right person, so we'd love to see your resume.)
There is nothing better than getting high-quality work done with a high-quality group of people.
Clutter is rapidly approaching 1.0 ... spent some time this weekend patching the remaining issue that was driving me crazy, hoping to get it in under the wire. Emmanuele may be owed beers.
It's gratifying that the new default home screen of Facebook looks a lot like Mugshot.org, a site some of us came up with at Red Hat. We coded Mugshot's personal-lifestream-thingy before Facebook's news feed came out and before FriendFeed came out. Not saying either site copied us, but it's still nice to know at least our idea was good (even though there were lots of reasons we weren't the ones to get anywhere with it).
It used to be that the cool kids claimed to have listened to R.E.M. before Document (these are cool kids 20 years ago, I guess). I was listening to mutual fund manager John Hussman before he showed up on fivethirtyeight.com and in the Wall Street Journal. Am I a cool kid? Ha.
New York Times tries to explain why people get distracted by small(er) things like millions in AIG bonuses, while missing large things, like billions to bail out AIG's bondholders. Possible Occam's razor answer: most people don't know what a bond is, but they know what a bonus is.
Some spammer selling Hewlett-Packard products took over my Twitter account "hp", and now Twitter seems to have deleted it - and allowed someone else to grab the account. I emailed Twitter a while back asking them to fix it but all I got was an autoresponse saying "we've taken two weeks and still haven't gotten back to you, let us know if you still need help." I rarely twittered anything anyway, but ... lame.
Funniest thing today was a random snark on Ayn Rand that Miguel reposted.
Why is it reasonable to pretend that socialist or libertarian principles can guide a decision between 35% and 39.6% tax rates? Sure, 5% vs. 80% is an ideological debate. But 35% vs. 39.6% is a pragmatic debate. The vast majority of American voters agree on the main government spending items - social security, medicare, defense - that eat up most of our tax revenues. Controversial spending items are too small to affect the tax rate much. And the vast majority of Americans are opposed to 50-60% tax rates. We're going to stay right about where we are because big tax hikes and big spending cuts are both third rails. Here's some historical perspective; tax rates are much more stable over the last few decades than they have been over the last century.
Amy discovered Cake Wrecks (scroll past Sunday, they have non-wrecks on Sunday). Awesome.

litl at summit this weekend

Thu, 09 Oct 2008 09:58 -0400

For GNOME summit attendees we're buying some coffee, dessert, and snacks on Saturday afternoon from a local French restaurant. litl's founder John Chuang plans to drop by as well to meet people and learn.

This isn't a talk or a big announcement (litl will still be in stealth mode post-summit) - the idea is to say thanks to the developers behind GNOME and introduce ourselves.

The litl devel team will be around all weekend, doing some hacking. As you may have guessed we have quite a bit of code written in JavaScript, with a C/C++ core. We've invested a few weeks splitting the JavaScript/gobject-introspection bindings out of our codebase and rebasing to newer gobject-introspection. By Saturday we hope to have it compiling and posted, so we can hack on combining our work with Alex's gscript API among other things. We might also sign up for a technical talk about the bindings if people are interested in learning more.

Why the credit crisis matters

Tue, 30 Sep 2008 23:58 -0400

In an effort to outdo other software developers in nerdiness, I read a blog called Accrued Interest, subtitle "Come for the analysis and research on the U.S. Bond market. Stay for the geeky Star Wars references."

Accrued Interest has a nice explanation of how the credit crisis affects everyone.

Here's another good one from the New York Times.

A less ideological health plan

Mon, 29 Sep 2008 19:45 -0400

I've been wondering for a while why nobody contrasts Obama's health care proposal with some of the past Democratic proposals. It's a market-based, incremental proposal, that makes use of existing programs and existing insurance companies.

Today Obama's campaign has a new ad making this point, so I guess I'm on their wavelength. After my What happened to John McCain? post, the next day Obama ripped that off, too, for a negative ad.

Finally fixing health care would make a huge difference for quite a few people I know. It may be the single most consequential thing the government could do.

If you watched the debates, see factcheck.org's coverage. They ding both candidates on multiple claims.

Yikes

Mon, 29 Sep 2008 18:15 -0400

Worst stock plunge in 20 years.

Here's a relevant post from last year, when the stock plunge was much less dramatic.

Lessons for the next bull market (yes there will be one eventually):

Low interest rates and high liquidity do not mean stock prices will stay up. Those things are indicators of insufficient risk premiums, not indicators of permanent demand for assets.
If companies have 2x their long-run average profit margins, something unsustainable is going on. Abnormal profits don't last forever in (more or less) free market systems, due to competition and the business cycle. Anytime you read an article that mentions a P/E ratio, without discussing normalized margins, run away in fear. (The inverse goes as we most likely head into recession. P/E ratios will become misleadingly high, while in the bull market they were misleadingly low.)

Tax policy

Sun, 14 Sep 2008 12:15 -0400

Nice, factual, 1-chart explanation of the competing tax proposals. (Thanks Luis.)

Barack Obama's policy proposals.

What happened to John McCain?

Sat, 13 Sep 2008 12:00 -0400

McCain was always the respectable Republican, who opposed tax cuts without spending cuts, did not deny environmental threats, had a reasonable approach to immigration, and was against the scorched-earth Rove-style politics. Somehow he has reversed himself on all of these things while running for president. It's not even clear he still opposes torture (see this vote, for example).

I don't consider flip-flopping in itself to be bad; the famous Keynes quote is, "When the facts change, I change my mind. What do you do, sir?"

But the facts haven't changed, McCain has. Read this interview with Time magazine - cringe-inducing. I just feel bad for him.

McCain 2008 is so unlike McCain 2000, it's very hard to understand what's going on. I'm not willing to write this one off as "politics" because it feels unusual and strange.

On top of McCain's personality-ectomy, since just before the convention, his campaign has kicked into a series of frivolous distractions. The New York Times has an article and an op-ed about it, but if you're a New York Times hater, factcheck.org is excellent and calls both candidates on their inaccuracies. factcheck.org doesn't try to make a subjective judgment about which candidate distorts the most, but read the last couple months of fact checks on each of them and see what you think.

On at least two substantive policy matters, taxes and health care, the candidates are failing to have a real discussion - McCain has been repeating misleading talking points instead of engaging with and debating the actual proposals.

And Sarah Palin - in her first interview, she says this with a straight face:

What insight into Russian actions, particularly in the last couple of weeks, does the proximity of the state give you?

They’re our next-door neighbors. And you can actually see Russia from land here in Alaska. From an island in Alaska.

Come on. She's applying for a serious job.

While a hollywood actor mocking creationism is not going to win over any swing voters, I have to agree with the "Disney movie" part of the sentiment.

To combat this, before announcing Palin's nomination, the McCain campaign seems to have invented a narrative that "they" are attacking her family, attacking her because she's a woman, blah blah. Even though Obama did the opposite and condemned attacking her family, McCain stuck to the talking points. And though it took his campaign a couple weeks, they found this "lipstick on a pig" comment they could use as "evidence" for the talking points they'd already been pushing. Obama responded very well - emphasizing how ridiculous it is to play games given the weight of the substantive issues at stake.

If your attention span is too short to watch Obama's serious responses, Jon Stewart takes apart the pundits on Palin in a more humorous way.

It makes me sad. When McCain won the nomination, though I was not going to vote for him, I was happy that we would have two people running with competence and integrity well above the politician average, and well above most of the others in the primaries. It felt somewhat "can't lose" - the upgrade from Bush would be significant either way. McCain 2008 is still better than Bush (it's not a high bar). But "new McCain" has been very disappointing compared to McCain 2000.

Mike Evans reposted a nice case for Obama from Marc Andreessen. Unfortunately, where the majority of young people and "knowledge workers" see competence, a lot of swing voters seem to see something negative.

There are serious problems facing the United States, especially the US economy, and the economic problems ripple through national security and other issues. Our economic problems have some straightforward answers: simplify the tax code, make it a bit more progressive so we have a solid base of consumers to buy stuff, reduce lost productivity and inflation caused by a broken approach to health care, and - the big one - address the redistribution of wealth and power from the US to other countries created by our oil dependency.

One of the presidential candidates wants to talk about credible solutions to these problems, and one of them wants to talk about distractions.

Synchronous IO Never OK

Sun, 07 Sep 2008 14:45 -0400

"If synchronous IO becomes a problem, it can be made asynchronous later." Tempting to imagine that some operations on local files are "fast enough" to implement with synchronous IO. "Premature optimization is the root of all evil," right?

Wishful thinking. Async vs. sync IO is not a performance issue (in fact synchronous IO can be faster). It is a structural or qualitative issue, a design issue. Sync IO is guilty until proven innocent.

My new rule is: any code that will be blocking during a user-interactive main loop must run in a known-short time. The task must be known short, not indefinite-but-often-short.

Local file access could be short, but is not known short. Another common culprit: synchronous D-Bus calls.

I've seen so many rewrites from synchronous to asynchronous IO over the years - it is almost 100% likely that anything blocking the main loop eventually comes to be seen as a bug. There's near-zero chance it's really OK to use those nice, easy-to-program blocking D-Bus calls, or that nice, simple g_file_get_contents(). Even the harmless looking g_file_test() can bite.

These APIs exist only to tempt you. They are the dark path.

(Note: by "asynchronous" I mean "the main loop is not blocking, for example because IO is in its own thread" - I don't mean special AIO system calls.)

Here's the core issue: the UI's main loop needs to wake up at 30-60 frames per second to do animations and repaints, and it should respond to user input with similar speed. It needs to do this consistently (think real-time-like), not "on average" - if there are big outliers, like an occasional quarter-second delay in frame rate, the app will feel sluggish.

"Local" IO can be slower than you think; the firefox fsync problem shows one extreme case (the whole kernel gets bogged down, not just the IO operation), but think about network file shares, or large file sizes, or systems under load. Even mild manifestations of these issues are visible in decreased user responsiveness and animation smoothness.

It's very common for supposedly fast IO operations to end up batched together, creating a long delay at once; for example, calling stat() on everything in a directory, or queuing a bunch of idle handlers that all happen to kick in at once.

Another Firefox example: try uploading a bunch of photos to Flickr in one go. Ouch. The files are local, but ... there are a lot of them and they are big. Pretty sure Firefox uses synchronous IO here.

Enough "this local IO should be fast enough" laziness scattered around an app creates distributed bloat that's hard to pin down and doesn't show up very well in profiles because it depends on external circumstances.

Not only is it hard to find later, synchronous code is hard to fix - it means rewriting.

Bottom line: don't write this code in the first place. Using async APIs is not premature optimization, it's correct design.

Embeddable languages

Mon, 25 Aug 2008 18:45 -0400

Chris mentions JavaScript embedding using SpiderMonkey.

When GNOME first started out, there was much talk of using Guile as an embedded language. The concept was similar to the architecture of Emacs, Firefox, or all the games using Lua (such as World of Warcraft).

I would define an "embedded" language as one that doesn't come with a platform. It's small and easy to glue together with an app written in C or C++. It uses the same platform as the app. If the platform has an introspection capability like XPCOM or GObject-Introspection, then the platform can be exported to the embedded language in an automated way.

At some point, and I was one of the people to blame for this, we changed the focus of the Linux language discussion to grand unified schemes involving component technologies and virtual machines. The result is that too much desktop stuff is still written in C.

A "C/C++ core plus embeddable language" architecture for an application has many pragmatic advantages.

Higher layers of code such as UI code are so much easier in a dynamic "scripting" language.
Proven to be practical and scalable for huge apps such as Firefox and World of Warcraft, even with dead-simple languages such as JavaScript and Lua.
No platform silos. In a GNOME app, the embeddable language uses GIO, GTK+, GStreamer, Clutter, or whatever you are using - it does not have its own http stack and everything else.
You can choose to write each part of the app in the easiest language; some stuff is easier in C/C++.
Sandboxable / well-modularized. The interpreter is generally an object that can be destroyed; no heavy language runtime or VM takes over the entire app.
Embeddable languages don't seem to create a huge jump in memory footprint, for whatever reason, so they are still sensible for "swarm of many processes" environments like GNOME and KDE that multiply any per-process overhead by a significant factor.

(There are advantages to the heavier full-platform languages also, but I'm guessing everyone knows them already so I won't make a list.)

JavaScript is a great choice for an embedded language. I did not realize at first that SpiderMonkey has no dependencies; it is not tied up with the rest of the Mozilla platform (XPCOM, etc.), it's completely standalone. And most programmers understand at least some JavaScript already.

There are some problems right now, for example there's no standard packaging and .pc file for jsapi.h/libmozjs.so, so it's hard to make an app build on multiple distributions unless you suck in an internal copy of the library.

It's necessary to add a couple things to JavaScript before it's truly usable; most notably, some kind of module system. I'm not sure SpiderMonkey itself should have that though - it starts to define a platform and make the language "not embedded". Maybe modules are introduced by higher layers (in fact xulrunner adds a module system if you use the whole xulrunner platform).

I don't mean this as a SpiderMonkey advocacy post, though we're using it successfully at work. Other options are the WebKit JavaScript engine, Guile, Lua, Tcl, etc.

My main point: the "C/C++ core plus embedded scripting" architecture is a very smart way to write an app, and doesn't have some of the real downsides of heavier platforms.

Return on Equity

Sun, 03 Aug 2008 14:45 -0400

Most people know about the price/earnings ratio. If you want to learn one more thing about financial numbers, return on equity could be a good choice.

Return on equity is annual profit divided by net worth. If you have $100 in net worth, and make $10 in annual profit, then you are making 10% ROE. If you think you can make a 15% return elsewhere, you could sell your assets to net $100 in cash, and invest in the 15% return elsewhere.

Reinvestment

Depending on the nature of the business, maybe you can add more net worth and make more profit. If you're making $10 on your $100 tied up, maybe you can add another $100 and now make $20. Still 10% ROE, but more money made.

This is supposed to determine what companies do with their profits. If they can reinvest into the business (or at least, some business) at a decent ROE, then they should do so. But if they can't, they should be paying the profits out as dividends their stockholders can invest elsewhere.

Say you propose a project for your company to undertake. The key questions might be the return on equity, and how much equity can be invested. (Making 500% returns on twenty bucks might not be very interesting for a large company.)

Where do earnings come from?

Return on equity can be broken down to understand why a business activity is making money. An equation called the DuPont system tries to do this.

The DuPont system is:

  ROE = 
    (Margin, how much of each sale is profit) x
    (Asset Turnover, how quickly inventory is sold) x
    (Financial Leverage, how much did we borrow)

Higher margin means more profit per dollar of stuff sold
Higher asset turnover means selling stuff more quickly
Higher financial leverage gives you more stuff to sell or more productive assets for the same net worth - you're buying machinery or inventory with a loan

Different kinds of business make money in different ways, even in the same industry. For example, some few-months-old numbers for Dell and Apple (numbers don't quite add up due to rounding error; numbers are from Morningstar):

Dell: 5% margin x 2.0 turnover x 4.4 leverage = 43.8% ROE
Apple: 15.4% margin x 0.9 turnover x 1.8 leverage = 24.2% ROE

Not surprising. Dell makes money by selling a lot of stuff quickly for a low profit, using more borrowed money. Apple makes money by selling more slowly, but for higher profits. Both are workable models, neither one is wrong. Both companies are single-minded about their approach; Dell has focused on increasing turnover rather than increasing margin, and the opposite for Apple.

Leverage is the most dangerous way to make money, because if the margin flips negative the ROE goes negative. Banks make their money on leverage, and they are supposed to be conservative about ensuring the margin stays positive, for example by only making sound loans to people who can pay them back.

The worst kind of business is high leverage with a low, but unstable, margin. Example (other than subprime lending): airlines.

American Airlines a few months ago: 2.2% margin x 0.8 turnover x 10.8 leverage = 19% ROE
American Airlines now: -7.1% margin x 0.8 turnover * 17.42 leverage = -98% ROE

Airlines are always about to go bankrupt. The margin depends on ticket prices (which they can't raise very much without losing sales), and fuel costs (which they don't control and which bounce around wildly). They can't control the margin, and on top of that the margin is low so requires high leverage to make the ROE worthwhile.

Perhaps this explains why airlines are awful to customers. They can't afford to spend any money.

Applications

Here's what a business activity should not be about: growing revenue for its own sake. Often it's possible to grow revenue by reinvesting in a business (say, hiring more salespeople, building more manufacturing capacity, running ads) ... but the ROE may be too low. If customers don't really want the increased sales enough to pay for them, then margins are low or even negative. At that point it would be better to invest in an index fund, or a savings account, rather than in the business. It doesn't make sense to spend a bunch of money selling stuff people don't want.

ROE determines what sort of price/earnings ratio is acceptable, because (in theory) the earnings will be reinvested at the ROE. If an investment pays 10% per year reinvested at 10%, that is worth a lot more than 10% per year reinvested at 2%. So the stock price should be higher relative to earnings.

I find it helpful to flip price/earnings over (earnings yield). It gives me much more pause to invest in something yielding 2% than something with a P/E of 50. Higher P/E almost sounds good - it's higher, right? Maybe this explains the tech bubble.

Both ROE and P/E require all kinds of adjustments and normalizations to remove accounting artifacts before they mean all that much. That's one reason it's futile trying to buy stocks if you aren't a qualified analyst (or relying on one).

As concepts, though, these ratios should help you decide what makes sense for the businesses you're involved with as an employee or owner. Is your company more focused on turnover or on margin, for example? Is your new project proposal profitable enough - would you personally invest in it instead of a mutual fund, if it were your money?