Coding Sanity

Marcus Ranum has an article titled  The Six Dumbest Ideas in Computer Security. I strongly suggest that anyone interested in security read it, and his other articles too. It's quite an old article, but someone put it on proggit and it made me remember it. In fact, I'd say that this article was seminal in guiding my undertstanding of computer security. For those of you too lazy to click on links, a swift summary:

1. Permitting things by default instead of denying things by default. Used to be very common in firewalls.
   
2. Focusing on stopping bad things rather than only allowing known good things, a special case of #1, but still very common in anti-viruses and operating systems.
3. Catch holes by fixing hacks as they become known rather than hardening the application.
4. Hacking is cool.
5. Educating users.
6. Doing something is always better than not doing something. Think about it this way, do you really want to be running bleeding edge software for your firewall?

A point he makes about #3 and #5 is that if these worked they would have worked by now. IE would be the most secure browser on the planet and users wouldn't click on attachments anymore.

Do yourself a favour, go and read the original article. It's well worth it. 

I recently received a chain email from a friend where the original author was complaining bitterly about the massive electricity price hikes being mooted by Eskom (our state-owned electricity company) in order to pay for their build programme. The author felt that since it was Eskom and government who were asleep at the switch and allowed the power situation to reach its current dire state, that the electricity price should not be raised. Instead, the author suggested that the power stations be built using the surplus that government has accumulated over the past few years.

It's in a slightly histrionic tone, but I think many people would sympathize with the author's points, certainly the 278 people who had signed it by the time it got to me did. The Eskom situation easily inflames people since we're not only being slammed with a price hike, but also rolling blackouts, and to top it all off, a large amount of the problem seems to be from fat cats keeping their eye on profitability rather than on long-term viability. To top it all off, the stories about Eskom supplying Zimbabwe with free electricity added the outrage over our government's dithering on that country to the issue. What I'd like to do is look at things from a less emotional point of view. I'm incredibly lucky in having some inside information about this, since certain colleagues have done modeling on this issue. I don't unfortunately have the raw figures, so I'm going to have to try and remember them as best I can.

This is the situation we find ourselves in

The first thing to remember is that complaining about Eskom or government or whoever may make us feel a lot better, but it won't actually solve anything. What has to be dealt with is the situation that exists right now. What this boils down to is reserve margin. From the Wikipedia page about Eskom I see that their overall capacity is about 39,046 MWe. So, that's the amount they could produce in a perfect world. However, the world is not perfect, and there are situations where you have to take a power station offline in order to maintain it, and of course you have unplanned outages as well. So, the idea is that the total amount you can produce exceeds what you need by a margin, the reserve margin. It is this that has been allowed to drop too low. Eskom have enough generating capacity, in theory, to supply all of South Africa's needs. However, to do that, they'd have to stop maintaining the power stations, and we can all guess where that would lead us. The ideal reserve margin seems to be at about 15%, and ours is about 6-8% right now.

Now an important point is that this reserve margin is over our peak usage. The usage of electricity during the day is similar to the graph you see here. Electricity usage is low at night, climbs rapidly in the morning, dips a bit during the course of the day, and then climbs in the evening before settling down again.

So our problem is that our usage is rising above the ideal reserve capacity during the day, and this is why Eskom is being forced to have these blackouts (I refuse to call it load shedding, its bloody rolling blackouts, not some saccharine platitude).

So, what can we do about this situation? Well, two main approaches are fairly obvious:

1. Increase supply

2. Decrease demand

Let's look at them in a bit more detail.

Increase Supply

Well, this is nice and simple it would seem: build more power stations. All well and done, but it costs money to build power stations. More importantly, the big ones take a long time to build. The following are some of the things we can do:

Buy more power from neighboring countries

Unfortunately we're by far the biggest producer and consumer of electricity in the region, so the amount of power we can buy is limited. Eskom recently upped the amount we get from the Cahora Bassa hydroelectric station by 250MWe if I remember correctly. It will help, but it's a drop in the ocean.

Build more power stations

There are a few main kinds of power stations we can build. Coal powered stations are huge and take a long time to build. On the upside they produce vast amounts of power very cheaply. Eskom are building two huge coal powered stations called Bravo and Medupi which will start coming online in about 2012. They are also de-mothballing another three, but they're not nearly as big. Another main type of station they can build is gas turbine. These take far less time to build, but they also produce far less. Unfortunately they also cost vastly more to run as well, so the electricity from them is very expensive. Normally, because they can be switched on and shut down fairly easily, they are used only to deal with demand spikes. Hydroelectric has the advantage of being fairly green and fairly cheap. Unfortunately we live in an arid country, so the amount of these we can build is limited. Finally, we have nuclear stations. These make an awful lot of sense, the electricity from them is fairly cheap, like coal, they're "base load" stations. But, like coal stations, they take a long time to build and are very very expensive.

"Green" electricity production such as wind farms and solar is also a possibility, but tends to be fairly limited in the amount of power they can produce as well. Plus, they do tend to be expensive. One of the nice things about them is that they can start coming online fairly swiftly, and every little bit helps.

So, it looks like a good strategy would be to build some green producers and gas turbines, and boost the amount of coal and nuclear stations in the long term. This is in fact exactly what Eskom are proposing to do. Again, I might have these figures a bit off, but they plan on spending about R1, 3 trillion over the next 10-15 years on building new capacity. This is an awful lot of money, and it has to come from somewhere.

Encourage cogeneration

Cogeneration is quite interesting. There are quite a few industries in South Africa that produce an enormous amount of heat from their processes. What happens with cogeneration is that this heat is channeled into turbines which then provide electricity to the national grid. Now, a company is not going to do this out of the goodness of their hearts, they're going to expect to be paid for the electricity they supply. Unfortunately, the low price it is at right now is not enough to make this feasible. However, the increases Eskom is proposing will move us into a position where cogeneration will become viable. The amount of cogeneration capacity may be as high as 5,000 MWe, basically the same as a huge coal-fired power station.

Encourage independent power producers

This is in fact what government was hoping for, why they didn't want Eskom to build more power stations. They were hoping that companies would come to South Africa and build power stations. Unfortunately, for some reason, companies did not flock to compete in an environment where their only customer set the price, and more importantly the price was too low for anyone to compete with. Building a power station is expensive, and you have to recoup your capital costs somehow. Eskom didn't have to, since they already had the power stations, and had pretty much paid them all off. Anyone else coming onto the market would have to compete in an environment where they would be unlikely to be able to make a profit. Interestingly enough, the price increases that Eskom is proposing will likely make potential independent power producers interested. It would now be possible to compete. This is largely because Eskom is now undergoing the same kinds of massive capital costs an independent producer would have to go through.

Encourage energy-negative houses

In some countries, the power utilities are required to buy back any power you produce at home. So, if you have solar panels and your house is producing more power than it consumes, then you get paid back. This is a quite nice idea for encouraging energy efficiency, but it is unlikely to be terribly effective in South Africa. Even with Eskom's price increases, electricity would still not be so expensive as to make this profitable for the individual concerned. In any case, it is feasible for only relatively well off households, which is a small percentage of the total.

Decrease Demand

This one is fairly obvious, since Eskom are using rolling blackouts as a mechanism to force decreased demand. What are some other alternatives?

Encourage energy efficiency in households and offices

This is fairly obvious. Switching to energy efficient light bulbs can produce a marked saving in power, geyser blankets and timers similarly help. More importantly, timers can be used to drop demand during times of peak usage. If every single person who has a geyser switched it off from 6am to 7pm, they wouldn't greatly inconvenience themselves and it would make a significant reduction to peak load. Switching from electricity to gas would probably help a bit, but not as much as you'd think, largely because you don't really cook during peak times. Tumble dryers and air conditioners are proverbially profligate in their energy usage. You could look at solar water geysers, but I think you'll find that a timer is just as effective, and much cheaper.

Apparently up to 15% of your house's energy usage is consumed by appliances which are plugged in, but "off" in a standby mode. Your TV, hi-fi, cell phone charger you left plugged in and so forth. We could encourage people to unplug these.

However, this raises a problem. All these ideas either cost the consumer money or they are a pain in the neck. Why would people do these things? South Africa is remarkably wasteful when it comes to electricity, a hangover from decades of cheap power. I've heard numerous suggestions such as punishing people who use "too much" with punitive fines. However, who decides what's too much? What if I have 6 people in my house and you have one in yours? It becomes very, very tricky. However, capitalism has a nice and simple way of encouraging people to use less of a thing whilst at the same time punishing those who use "too much". It's called raising prices.

Unfortunately, people can easily become habituated to small changes, so to force people's behavior to fundamentally shift you have to shock them, and a big increase is likely to accomplish that, whilst numerous small ones will probably not.

Shut down industries

This is not a good one. Shutting down mines and aluminum smelters saves power, but it harms the economy as well. Badly. In any case, as with normal consumers, the best way to get industry to use less power is to charge them more for it. A lot more. This will lead to inflation, true, but it is far better to see some inflation rather than entire sectors of our economy being brought to a standstill. Unfortunately our Reserve Bank has an itchy trigger finger when it comes to inflation and is pursuing a punitive interest rate policy to keep it in check. I think we will have to let the Bank allow the inflation caused by Eskom's increases to work their way through the economy without raising interest rates, even if it does mean some spiking in inflation. If not, the Bank may all but shut down the economy. This would help conserve power, but not in a way any of us are likely to enjoy.

Punitive blackouts

This is what we have right now, where rolling blackouts happens no matter what, unless you cut your usage by 10%. The only problem is that Eskom and the municipalities are not geared to blackout small areas, so it's groups of suburbs that go down together. Ideally, in my opinion, they should try and get to a point where a blackout unit is about 10-20 houses. Then it becomes feasible for communities to encourage and work towards energy efficiency in their area. If they do so, and cut their usage by 10%, they stop getting blacked out. Right now, it's not really practical to lead a community effort to cut usage by 10% across all of Johannesburg, but a street, now that's doable. When you're sitting in the dark and the guys in the next street have lights, maybe you'd consider changing your light bulbs or putting in other energy saving mechanisms.

Denying access to power

This is another approach that Eskom are using; they're simply refusing to provide power to new developments. The long term effects of such a strategy will be horrific though. What they're essentially doing is stopping most new fixed investment in South Africa, investment we critically need for our economy to grow. This cannot continue for very long without causing a massive problem.

Changing the peak structure

This is one I find personally interesting. The idea here is not so much to conserve power, but rather to shift it, to flatten out the peaks of demand by moving that demand to the troughs. An obvious way of trying this is to charge more for peak times, just as the cell phone operators do, maybe even give "free" power away in off-peak times. Again, the difference would have to be big enough to make people sit up and take notice.

Another very interesting proposal has to do with time zones. The idea here is that we split South Africa into two time zones. I'd guess that Cape Town, Port Elizabeth and Bloemfontein would fall into the one time zone, and Johannesburg and Durban into the other. This morning the sun rose at 07:11 in Cape Town and at 06:17 in Durban. So, if South Africa is in two time zones, an hour apart, that usage pattern above would be flattened out a bit, and widened. I personally find this idea fascinating and well worth the attempt. Changing time zones would not cause too much inconvenience in any case. For goodness sakes, have you ever tried to get hold of a Capetonian before 9am? Not a chance. So, this would just formalize that

Going forward

Quite simply, the price hikes that are being proposed are important and necessary. There is the worry about how they will affect the poor, obviously, and how the inflation created would similarly add pressure to them. However, the poor would be worse affected by our economy tanking and their ranks being swollen with newly unemployed. Right now, the fundamentals of our economy are in pretty good shape. Besides our trade deficit which is a bit worrying, the only other cloud is this power problem. If we solve it, then there's no reason our economy can't carry on growing. Of course, that would increase demand for electricity even more. You win some and lose some.

Anyway, I digress, the point is that there is a lot we can do to increase supply and most of it is being done or planned for. However, to increase supply enough will require a significant increase in electricity prices; effectively they are going to have to double over the next two years. The problem is that the supply increases will largely take a long time to come on stream; we simply can't build the power stations quickly enough. Therefore, for the short and medium term we have to look at constraining demand. Blackouts and stopping new developments will quite simply bring our economy to a screeching halt. We're doing these now, because we don't have any choice, but we must aim at getting rid of these measures at the earliest possible time. That means consumers, business and industry coming to the party on saving power. Eskom and government's pleas to consumers and business have not had much effect on people's behavior.

Personally, I think part of the reason for that is the anger at them for letting this situation develop in the first place, there's a sense that it's "their problem". Unfortunately, it's not, it's our problem, and will remain so for some time to come. A lot of people are understandably unhappy about the proposed price increases and see it as being punished for someone else's mistakes. Certainly there is an element of that, but think about it this way. For the last decade or so, you've had electricity at a discount. Those discounts are coming to an end. Sure, it's not nice, and it would be easier to swallow if the lights stayed on the whole time, but they were at a discount, and prices are just moving to where they should have been. Eskom and governments mismanagement extended to not increasing prices as much and as fast as they should have been. So in a sense we all benefited from their mistake in the past, and are suffering for it in the present.

A final thought. Whenever I have issues that afflict me, work or personal, I always try and think about two things:

• What could I have done differently?
• What can I do in the future?

It very rarely helps matters to worry about who to blame, all that does is distract you from the solutions. In this case, yes, there are definitely people to blame, and I doubt there's much many of us could have done in the past, since we didn't even know about this problem. However, we can look to the future. By conserving power, especially at peak times, even if all you do is cook a little later or install a geyser timer you can actually make an important difference. More importantly, by just being a little more energy conscious you will be planning for your future.

I assure you, this power issue is the tip of the iceberg. Energy costs are rising worldwide, and this trend is going to continue. We will look back on this power cost increase in a decade and laugh at how little power cost in 2010.

There seems to be a serious issue with Microsoft lately, and it seems to be a strategic and executive issue. They have, for a couple of years now, been desperately running around trying to own every technology market they're not in whilst at the same time allowing Windows, their core product to "collapse". Witness their doomed attempt to take on the iPod with the Zune. The question has to be asked, why? I mean, I can understand why Creative or Sony or another portable media company might try and rush out ill-advised competitors to the iPod, it's a major part of their business after all. But why for goodness sakes would Microsoft decide to spend a fortune taking on an established incumbent? Especially after waiting years for them to cement their lead before getting involved?

Forget the numerous complaints about the Zune itself; I'm just trying to understand their strategy here.

Let's not forget their decision, many years ago, to take on the games consoles. Well, that seems to have worked out okay, although it sure cost them a ton of money. Now, why did they do that? Well, the worry was that consoles might supplant PC's, a reasonable concern I guess, unlikely, but possible. They sure couldn't have done it to sell software, since MS games were few and far between at the time.

So now we get onto their biggest strategic blunder; their desperate attempts to unseat Google as the king of search. It seems that Ballmer and Co are so terrified of Google and the *gasp* massive advertising revenues they earn that they are literally willing to bet the entire company on a chance, not a certainty, but a chance that they can eat into this revenue stream.

Financials 

Let's look at some figures, to be precise the 2007 Q4 earnings reports from Microsoft and Google:

Market Cap

	Microsoft	Google	Difference
Total Assets	$63,171,000,000	$25,335,806,000	$37,835,194,000
Net Income	$3,035,000,000	$1,030,716,000	$2,004,284,000
$266,180,000,000	$145,440,000,000	$120,740,000,000

So, Microsoft has assets of 2.5 times Google's, it's income is 3 times, and it's market cap is 1.8 times. Now, this is by no means a complete picture. For starters it is my opinion that Google is in fact significantly overvalued, however I'm certainly no expert. Up until fairly recently I'd also have said that Microsoft is undervalued, but I no longer think that. This is not because I think that the company as a whole is valued correctly, I just feel that the utterly idiotic actions recently demand a discount to be placed onto the Microsoft valuation.

Yahoo!

To be precise, the idiocies to which I am referring is the Microsoft desire to buy Yahoo come hell or high water. As a matter of interest, Yahoo has a market cap of a mere $37,860,000,000, meaning that a "perfect" merger that inflated Microsoft's market cap by Yahoo's would result in Microsoft gaining a mere 14%. Needless to say, no merger goes that easily. In this case you're merging two companies with incredibly different cultures and overlapping products, so the value to be gained from the deal is at a significant discount.

The question has to be asked: what will the Yahoo deal do for Microsoft as a company? The answer: not a helluva lot. I'd imagine most of the really good Yahoo developers will not be too keen on working for Microsoft, so we can expect a raft of defections. Then Microsoft will have to shut down and merge various divisions, paying massive retrenchment packages. The final result: an even more bloated and top-heavy Microsoft with a few niche internet products and one or two good ones (flikr for example).

Many years ago I worked at a large South African IT company with management who'd seriously stuffed up. It seemed that, in order to hide from the shareholders the magnitude of their mistakes, they engaged in acquisition after acquisition. You see, due to all the merger activity, it was incredibly difficult to compare one year's performance with another's, since the company profile had changed so much. Needless to say, the constant, poorly thought out acquisitions were part of the problem, and eventually resulted in the company all but collapsing.

What are they doing?

Given Steve Ballmer's Bush-esque charge for this Yahoo deal, ignoring overwhelming opinion that it's a bad idea and seemingly paying attention only to that tiny group of people who think it's a good idea, one has to ask what his motives are?

• Is he honestly convinced that such a deal will suddenly turn Microsoft into a Google-killer? If so, the shareholders need to swiftly find a CEO who actually has his feet planted in reality rather than cloud cuckoo land.
• Does he believe that advertising revenues are going to grow so huge as to turn Google into a Microsoft-killer, able to wipe out all of Microsoft's products at a whim? In this case, he needs some meds for his paranoia. He could also look at trimming down the massive bureaucracy that Microsoft has become, weeding out the time-servers and turning Microsoft lean and mean again. Steve needs to ask himself how successful Big Blue was in wiping out Microsoft back in the day.
• Is he hoping that this deal will obscure the upcoming financial disasters due to mismanagement of the Windows stack? Whilst this is my latest pet theory, it doesn't really hold water, unfortunately.

Basically, I'm concerned that Microsoft is making misstep after misstep, from Vista itself and the Vista-capable debacle, to the Zune, the hugely overpriced Facebook deal to Yahoo, and many, many others. As a .NET developer I have a vested interest in seeing Microsoft succeed. Sometimes that means chiding them for their mistakes. However, I am growing increasingly concerned about the direction Microsoft executives have been taking recently. If I was Google, and had to pay someone to destroy Microsoft, I'm not sure if I could get them to do a better job than the current rash of poor decisions.

My advice to MS

I wish Microsoft would stop panicking about every upstart industry or internet fad that comes along, and focus on their products. Get Windows to where it should be, and you can probably kiss Apple OS and Linux goodbye. Work with hardware manufacturers well enough and you could possibly kill off MacBook too, or at least relegate it to a niche market. Drop the Zune, let Apple keep that market, with enough care you can effectively turn them into a consumer electronics company like a Sony. In other words, make their flair for design other people's problem, not yours. Make a decent online-capable rich client Office, along with web versions, obviously not as full featured, and you can kill off any possible Google Documents challenges.

Forget advertising, it's a niche and fickle market. Let Google have it, there's already some rumblings of discontent amoungst advertisers around Google's opaque pricing strategies. Let them deal with that pain. Sure they're going to control search for a while to come, which means they will effectively be everyone's landing page. So what?

Oh, and the #1 strategy? Install an ad blocker in IE, and offer to switch it on when the users install IE. You know, something like "Adverts can slow down your browsing experience, and distract from the content on the page". Something like that. Make sure it blocks Google ads. Yeah, sure you're going to effectively de-monetise the web, but since your web efforts have been, ahem, well, not as successful as you'd like, who the hell cares? So half the internet companies subsisting only on advert revenue would shut down almost immediately, why the hell would this be your problem? Best of all, make the ad blocker easily replaceable and you probably wouldn't even have to worry about anti trust issues, especially if you gave up on competing on search.

A friend of mine is looking fo ASP.NET developers. He's basically looking for mid-level to more senior people, who know their way around the frameworks. The person would have to be a self-starter able to work without constant supervision.

If you're interested, his name is Peter Wright and you can get hold of him on (082) 610-0605.  

Dunno how many of you read John Lilly's rant about Apple pushing out their Safari browser via iTunes update the other day? Basically he was rightly upset that Apple were using what should be an update and patch delivery mechanism to subtly trick users into installing their Safari web browser. His main point was

Apple has made it incredibly easy — the default, even — for users to install ride along software that they didn’t ask for, and maybe didn’t want. This is wrong, and borders on malware distribution practices. 

I couldn't agree with him more. The issue is not so much that they deliver Safari via Apple Update, but that it's checked by default every time. The amusing thing is the usual strident defenses from Mac fanboys:

...You Windows users are not only paranoid, but so anti-Apple that your comments are hysterical...

Much to do about nothing! The problem here is clear: Mozilla’s CEO is intimidated by the competition.

Mountain out of a molehill… at least Safari is standards compliant.

I think the problem here is that Safari out performs both IE and FireFox. This is just sour grapes. I would like to hear more about the performance differences.

 What makes these vociferous justifications all the more poignant is the recent failure (again) of Apple in the PWN 2 OWN hacking contest. Now, for two years running, Apple has finished dead last due to security glitches in Safari, the very same Safari they are so eagerly trying to force down the throats of their iTunes customers.

Let me give you a brief synopsis of the hacking contest:

• Day 1: Three machines (Mac, Windows, and Linux) are placed on the network, and left for the entire day with no user interaction for contestants to attempt to crack. Prize for succeeding: the laptop you crack and $20,000.
• Day 2: The organisers use various default installed client-side applications. Basically the attackers can ask the organisers to open emails, or visit web sites, but not to open attachments or install programs. Prize: laptop plus $10,000.
  
• Day 3: Much more open, some popular third party apps will be installed and used. Prize: laptop plus $5,000.
  

The Mac fell 2 minutes into Day 2. As of this time neither the Windows or Linux machine have been compromised, with Day 3 still to go [Update - Vista fell 7 hours into Day 3]. The Slashdot thread on this is also illuminating, with the most coherent fanboy conspiracy theory the severely addled idea that the researchers wanted a Macbook more than the other laptops. Therefore, they decided not to attack the other machines and concentrated on the Mac, clearly ignoring the fact that $20,000 or $10,000 can buy you quite a few Macbooks. Another idea from the sheep was that the Mac is so secure that the hackers wanted the challenge of attacking it first. The mind seriously boggles, but I guess once you enter the Reality Distortion Field your mind goes blank and you start drooling like a slack-jawed idiot. One of the best responses I saw to one of the fanboys was this one:

Really? Because I see the Mac having come out as the clear loser in a head to head contest on a level playing field against the two biggest competitors it has in the laptop market. Seems pretty simple to me.

Now let me stop giggling and get back to my point, I'm sure I had one. Do I think that Mac is crap because of a Safari flaw? No, not at all. Besides my digression into the vagaries of fanboyism, all it says is that Safari is not very secure compared with IE7 and Firefox. In fact, it's so insecure that twice it's been the cause of Mac failure at this competition.

This is the application that Apple has decided to share with us in an underhanded and sneaky way. They are surreptitiously installing an application which provides known attack vectors into the operating system. More particularly they're supplying this to people who probably have more secure browsers already, such as Firefox or IE. Does this strike anybody as strangely familiar? (Hint: Sony rootkit)

Now sure, this is not at all to the same degree, and there's no guarantee that Safari on Windows is going to be as insecure as Safari on OSX, I just don't think Apple should be slipping this kind of stuff in on the sly is all.

Update

The Vista machine went down 7 hours into the third day due to an Adobe Flash vulnerability, leaving Ubuntu Linux running on a Sony Vaio as the last machine standing.

Now what this means is that the Mac was undone by an application that Apple pre-install on their computers, whilst Vista was undone by a third party application. Given Vista and IE7's vaunted security, it should not be possible to hijack the computer, no matter how many bugs there are in Flash. Nonetheless, Microsoft are not pushing out Flash (or even Silverlight for that matter) silently, and you have to choose to install it, a choice Apple is still deceitfully trying to take away from iTunes users.

Let me say it again, deviously installing software that opens your computer to attack vectors is pretty much the definition of malware. The difference between Apple’s tactics and Sony BMG’s is not a difference in principle, merely one of degree.

What's something everybody wants for their application, but very few people have the time to deliver? Performance. Let's face it, in most software projects, performance requirements are relegated to the very end of the project, when every knows they won't have the time to address them. In one sense this is a good thing, as one of my biggest bugbears is premature optimisation.

Premature optimization is the root of all evil.

- Hoare's Dictum, Sir Tony Hoare

When should I optimise?

Now keep in mind that by denigrating premature optimisation I am not saying that you should never think of performance when writing an application. Of course you should, especially when looking at your design. Sir Tony's' quote is all too often taken out of context as Randall Hyde argues. Good performance is most effectively obtained by thinking very carefully about your design up-front.

However, when you have the choice between an easy to write, slow algorithm and a difficult, fast one then chances are that you should probably write the easy one. This is not always true, of course. For example, you may know that the algorithm is being used on a critical path in your application, in which case you definitely should go for speed. Far too few people even keep these trade-offs in mind and go for one extreme or another; optimising code that will not be a bottleneck, or ignoring code that will. I often comment code that I know is slow with a // PERF: comment , so that I can go back to it later to improve it. The nice thing about this approach is that you can move the slow code into your unit tests in order to ensure that the results match your optimised code, since all too often bugs are introduced during optimisation.

The type of application you are writing matters a lot as well. Obviously, if you are developing a software product, you will want much tighter performance requirements from most of your code than if you're writing a bog-standard enterprise app. The reason? Well, in my experience almost all custom enterprise applications are IO-bound, and spend an awful lot of time waiting for the user or for results from the database. In such an application, your database design and tweaks will likely make far more impact than anything else you may do.

That said, what happens when, at the end of your project you find that your code is too slow to deliver to the customer? Well, apparently in one talk Rico Mariani said that the Ten Commandments of Performance are

1. Measure

2. Measure

3. Measure

4. Measure

5. Measure

6. Measure

7. Measure

8. Measure

9. Measure

10. Measure

Scott Kirkwood has some interesting arguments and counter arguments to premature optimisation:

[...] Now back to premature optimization. I think what they really want to say is that "Unnecessary optimization makes code that is unmanageable, buggy and late" and there's more:

• When a program has performance problems the programmer always knows which part of the code is slow...and is always wrong.
• Only through profiling do you really see where the performance issue is.
• You can waste a lot of time doing optimization that doesn't matter.
• Optimization can often make the code more obscure, and hard to maintain.
• Spending more time on optimization means you are spending less time on other things (like correctness and testing).

Well that's the theory. And here's some of my counter arguments:

• If a developer really enjoys what he is doing it wont take any "extra" time. In other words, taking time to optimize doesn't necessarily steal time from testing, more likely it steals time from surfing the web
• Every time a developer looks at the code for something to optimize, he's looking at the code! He understands (groks) it better and may fix more bugs.
• Encouraging developers to leave in code that they know is embarrassingly slow makes them a little less proud of their code, a little less enthusiastic about finding and fixing their bugs.
• Products have failed because in a review they mention that it took twice as long to load a document than the competitor (even though it was 2 seconds instead of 1)
• When you put the code in production and it's too slow, you may be able to fix it by profiling and optimizing, but then again, you may not - you may have to redesign it.

So, crack out your profiler (I'm a big fan of the ANTS Profiler from Red Gate) and measure and find your bottlenecks and optimise them. If you follow my approach with marking code you know to be slow, you might be surprised to find how rarely you are correct in your estimation of what code is a performance bottleneck.

When should I stop optimising?

Obviously, this is a big question to ask. Usually, you will get a few low-hanging fruit, a couple of optimisations that give you large performance benefits. After that, however, it will become more and more difficult to find good high-value optimisations. At this point most people stop optimising and ship the code. However, what if that's still not good enough? Well, let's think about faking performance.

John Maeda says

Often times, the perception of waiting less is just as effective as the actual fact of waiting less. For instance, an owner of a Porsche achieves the thrill of directness between translation of a slight tap on the acceleration pedal, to be manifest as an immediate burst of speed. Yet in any normal rush hour situation, a Porsche doesn't go any faster than a Hyundai. The Porsche owner, however, still derives pleasure from his or her perception that they are getting to work faster in a quantitatively faster machine. The visual and tactile semantics of the Porsche's cockpit all support the qualitative illusion that the driver is going faster than when he or she is sitting inside a Hyundai.

[...] The premise was when a user was presented with a task that required time for the computer to crunch on something, when a progress bar was shown, the user would perceive that the computer took less time to process versus having been shown no progress bar at all.

So, one of the easiest ways to fake performance is to slap in a BackgroundWorker component, put your expensive code in there, and report progress via a progress bar. Since you are adding not only an extra thread, but also more UI updates there is no doubt whatsoever that your code is less efficient, yet the user will perceive it as being more efficient.

Now, if even that's not enough, another approach is obviously to offload the processing to another machine. This is even better than the progress bar if the user does not need the results of the calculation right away, since they say what they want to happen and press the start button or whatever and can immediately begin working on something else. By offloading the processing, perhaps to a multiprocessor server, you are gaining a massive improvement in the users perception of the speed of your applications, as well as an improvement in the running time. The cost, obviously, is the work required to implement the handover as well as the hardware costs of the server.

Now, I am not advocating no optimisations at all, but I am trying to get across that sometimes these "faking it" approaches are easier and cheaper than extensive performance tweaking. Needless to say, sometimes, even with massive optimisation, you still need massive offloading capabilities. Just look at SETI@Home as an example.

Conclusion

So, keep performance very much in mind when designing your software, keep performance trade-offs in mind when writing it, keep difficulty and impact of optimisations in mind when profiling, and keep faking it mind when polishing your application.

Update

Jeff Atwood has a nice post on how changes to the File Copy progress bar made users see the copy as less efficient, even when it was in fact more accurate.

InfoWorld have a petition you can sign to ask Microsoft not to discontinue Windows XP on June 30 this year. More details can be found here.

In many respects, Vista is like the Windows Millennium Edition that was meant to replace Windows 98 in 2000 but caused more trouble than it was worth. At that time, Windows 2000 was promising but didn't support a lot of hardware, so users were stuck between two bad choices. Without admitting Millennium's failure, Microsoft quietly put Windows 98 back on the market until the fixed version of Windows 2000 (SP1) was available. Microsoft needs to do something like that again today.

I have finally decided to take the plunge. Last night I upgraded my Vista desktop machine to Windows XP, and this afternoon I will be doing the same to my laptop.

Look & Feel

Windows XP has quite a cartoony look and feel compared to the slick look of Aero Glass; this is mostly offset by the lack of strange screen artifacts caused by malfunctioning graphics code. You know, almost like static on the screen. This was a once or twice monthly occurrence on my laptop, and happened on my desktop whenever I logged in, and also whenever I played a 3D game after leaving Vista running for a couple of hours. I also miss the "orphaned windows" I got on Vista, dialog boxes that would not go away, in a sense they became part of the desktop, since you could drag a selection from within them, despite the fact that the Glass would render the selection below them. Such crazy graphics bugs appear to be a thing of the past.

Performance

Well, here there appears to be no contest. Windows XP is both faster and far more responsive. I no longer have the obligatory 1-minute system lock that happens whenever I log onto Vista, instead I can run applications as soon as I can click their icons. Not only that, but the applications start snappily too, rather than all waiting in some "I'm still starting up the OS" queue for 30 seconds or so before all starting at once. In addition, I have noticed that when performing complex tasks such as viewing large images, or updating large spreadsheets, instead of the whole operating system locking down for several seconds, it now just locks down the application I am working on, allowing me to <gasp> Alt-Tab to another application and work on that. I am thrilled that Microsoft decided to add preemptive multitasking to their operating system, and for this reason alone I would strongly urge you to upgrade to XP. With the amount of multi-core processors around today using a multitasking operating system like XP makes a world of difference.

In addition, numerous tasks that take a long time on Vista have been greatly speeded up. File copies are snappy and responsive, and pressing the Cancel button halfway through actually cancels the copy almost immediately, as opposed to having it lock up, and sometimes lock up the PC. In addition, a lot of work has gone into making deletes far more efficient, it appears that no more does the operating system scan every file to be deleted prior to wiping it, and instead just wipes out the NTFS trees involved, a far quicker operation. On my Vista machine I would often see a dialog box from some of my video codec's pop up when deleting, moving or copying videos. No more, now all that is involved is a byte transfer or NTFS operation.

Automatic Updates has also gone through a performance facelift in that it no longer hogs your bandwidth when you're surfing, a nice touch.

Device Support

XP comes with some impressive device support. In fact, every peripheral I've collected over the years works perfectly with it. Many have the device drivers preinstalled on XP, making their installation a snap, but for the rest it was easy to find device drivers on the Web. In addition I found the drivers quick and reliable, a far cry from the buggy, slow and sparse driver support in Vista. I'm glad to see that with their new flagship OS, Windows XP, Microsoft have finally learnt from the mistakes they made with the Vista launch. In addition, support for mobile devices seems to be significantly improved.

I've also found that XP seems much lighter on the hardware than Vista, when it's inactive the hard drive very rarely spins up, a major advantage for me, since I often sleep near my laptop. No longer do I have to try and ignore the continual hard drive drone, but can now sleep soundly just like my computer. I never did figure out exactly what Vista was doing with my hard drive the whole time, but I'm sure it degraded its lifespan with all that spinning.

Reliability

All I can say is "wow!" You can see that a lot of work has gone into making XP more reliable than its predecessor. The random program crashes, and hangs appear to be a thing of the past.

Internet Explorer 7 is much more reliable on XP as well, and has so far not crashed once whilst viewing GMail, when it used to do this several times a day. In addition, I can now actually close the thing down normally every time, instead of sometimes having to kill the process. Error collection seems to be far better as well. Instead of a dialog taking a minute or two to collect the information it needs, the dialog comes up and is ready to send error data almost immediately. I am sad to see the back of the Solutions tool though, it may have hardly ever delivered any valid solutions, especially for the standard random crashes, but at least you knew that something under your control was tracking that information. Please, Microsoft bring it back.

Speaking of which, I notice that the Reliability Report is also gone, again a sore loss, I really enjoyed charting the downward spiral of my Vista reliability, there were those occasional humps that got you all excited, and then the graph would continue its steady sojourn downwards. Of course, the fact that it only appeared to pay attention to a tiny fraction of the actual problems was a bit of an issue, but I'm sure they could have resolved that for the XP release. Ah well.

I also am pleased to note that Ctrl-Alt-Del does actually have an effect nowadays. Many times in Vista, I wished that they would make this more reliable so I could kill off the inevitable hanging Windows Explorer process (as a matter of fact, this is the situation I find myself in right now), in XP it actually does something as opposed to being part of the usual Vista eternal hang. Speaking of which, please excuse me for a few minutes, Windows Explorer has now been 100% hung for 5 minutes, despite my asking Vista to restart it, and despite me pushing Ctrl-Alt-Del several times over those 5 minutes. So I'm going to have to hard-reset my laptop. This process, by the way, is also something that amazingly seems to almost never be required in the clean and sparkling new XP.

Right, I'm back, thanks for being patient. I mentioned how much quicker you could start using programs from a boot in XP; I must admit that, appealing though that feature is, you won't actually find it that useful. XP almost never appears to require a reboot, so you hardly ever take advantage of a wonderful improvement like that, which otherwise would save you at least 15-20 minutes a day.

Gaming

This is another area where Microsoft has really excelled in Windows XP. Games are significantly more responsive, get much higher frame rates, and are far more reliable than in Vista. If you're a gamer, the upgrade to XP is mandatory. Whilst there are a few games that won't work as well in XP than in Vista, you'll find that on the whole XP supports almost all the games you'd want to play. In addition, it's vastly increased reliability means you'll spend much more time killing things than restarting, a welcome change I can assure you. You'll also find that non-X-Fi soundcards with EAX are much improved by their support in XP, which can really add a bit of excitement to your gaming experience.

Multimedia

Multimedia support on XP is vastly better than on Vista. Whilst content-creators had insisted on all sorts of intrusive features in Vista that made the multimedia experience a living hell for Microsoft users, thankfully with XP Microsoft were able to insist that their customers' needs came ahead of the content creators outdated business model. It's nice to see a corporation like Microsoft stand up to the cyber bullies at the MPAA and refuse to assume that its loyal customers are criminals. In any case, the DRM built into Vista was broken shortly after its release anyway.

Conclusion

To be honest there is only one conclusion to be made; Microsoft has really outdone themselves in delivering a brand new operating system that really excels in all the areas where Vista was sub-optimal. From my testing, discussions with friends and colleagues, and a review of the material out there on the web there seems to be no doubt whatsoever that that upgrade to XP is well worth the money. Microsoft can really pat themselves on the back for a job well done, delivering an operating system which is much faster and far more reliable than its predecessor. Anyone who thinks there are problems in the Microsoft Windows team need only point to this fantastic release and scoff loudly.

Well done Microsoft!

There seems to be some confusion in the sadeveloper forums about the Dispose pattern, so I thought I look at it in a bit of detail.

Deterministic Finalization

Deterministic Finalization is something that C++, Pascal and similar languages have. What this means is that when you decide that you're finished with something it is cleaned up immediately. The delete keyword in C++ is used for this purpose, it will call any destructors of the object, and then release the memory that the object uses. In the vast majority of cases the destructors do nothing important or do not even exist. In such cases all that happens is that memory is freed. However, it is very common for C++ programmers to forget to delete objects, resulting in memory leaks.

This is particularly the case when an object is used by multiple parts of the program. Since there is no clear "owner" for the object it is difficult to determine who should delete it, resulting in dangling pointers (pointers which point to objects which no longer exist) and memory leaks. To resolve this, many schemes have been attempted, such as auto_ptr<T> and the Boost Libraries shared_ptr<T>.

Well, given that memory is a much more abundant resource than it used to be, the question has to be asked as to why the deletion happens immediately when the object is no longer needed? Freeing up memory can be a quite expensive operation, and many optimizations in C++ involved putting the freeing code outside a loop rather than within it, where perhaps it made more sense, but was just too inefficient.

Garbage collected languages like VB.NET, C# and Java do not delete the object when it is no longer in use, they instead just stop pointing to it. Intermittently the garbage collection will run, which determines which objects are no longer pointed to and deletes them. This is a gross simplification, but good enough for our purposes. If you want more information I suggest you Read Jeffrey Richters articles Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework and Garbage Collection—Part 2: Automatic Memory Management in the Microsoft .NET Framework.

What this means is that the deleting of the object (it's finalization) no longer happens exactly when the object is no longer needed (i.e. is not deterministic). However, the reality is that with schemes such as auto_ptr finalization was pretty tricky to nail down. One would have to study the program very carefully indeed to work out where the object would be deleted. So whilst the finalization was deterministic, it was very difficult for a human to figure out.

Brian Harry, a developer at Microsoft says

[...] we have watched (and helped) hundreds of thousands of lines of code be written without deterministic finalization. Now I am convinced that substantial programs can be reasonably written and debugged without the system providing any automatic support. That said, I fully agree that it would be better if the system/language provided additional support. Without it, you must build the behavior into the contract of the objects (like calling the Dispose method).

Disposing

However, there is one big advantage of deterministic finalization. If the object holds some very expensive resource, such as a handle to a commonly used file, or a database connection, then when it is finalized these resources will be released immediately. In other words, the resource will be released as soon as it is no longer in use. With garbage collection however this is no longer the case, the resource will be released at some point after it is no longer in use.

Quite frankly, in many cases, this is just not good enough. So, the .NET Framework introduces the IDisposable interface. This interface has only one method Dispose, which, when called, is meant to release any expensive resource the object may have. IDisposable should only be implemented by objects with such expensive resources.

The Dispose method is called, usually by the creator of the object, when it is finished with the object. It then immediately frees up the expensive resource, meaning that all that remains of the object is it's memory contents, which will be tidied up automatically by the garbage collector. IDisposable is a way to get what is, to all intents and purposes, deterministic finalization in a garbage collected object.

Performance

Many purists will claim that deterministic finalization, where the expensive resources and the memory are disposed of at the same time, is much more efficient than a garbage collected solution. However, it appears that reality does not completely back up their claims. Garbage collection is very efficient at allocations, far more so than most implementations of malloc since the garbage collector defragments the heap from time to time. malloc does a similar operation, coalescing contiguous free blocks, but it cannot truly defragment because it cannot move in-use memory. Managed systems like .NET and Java can.

So, whilst malloc and it's ilk face a gradually declining memory performance, managed environments have very quick memory performance, coupled with periods of extreme slowness (no execution whatsoever) during a garbage collection. .NET uses generational garbage collection to help speed this process up. Also, keep in mind that unless there is significant memory pressure, the garbage collector will tend to run when the app is largely idle anyway. From wikipedia:

[...] the garbage collector allows the runtime system to amortize allocation and deallocation operations in a potentially advantageous fashion

So in a best-case scenario (large memory, periods of idle) a garbage collected system can spend much less time dealing with memory than a non-garbage collected one. In a worst-case scenario (small memory, 100% CPU), the garbage collector would be likely be slightly slower than a non-garbage collected environment.

However, garbage collection, since it is not deterministic, cannot be easily used in embedded or real-time systems.

Conclusion

With garbage collections benefits, the only real penalty is this deterministic finalization thing, and IDisposable gives us something that approximates this. However, there is a big problem with this: what if the person using your class doesn't call Dispose? Then the clean-up of the resources doesn't happen. So we need a way to ensure that it will always happen. We use the appropriately named Finalize method in .NET. I will address this in my next article.

Next: The Dispose Pattern - Finalizers and Dispose

I saw this absolutely hilarious comp.compression thread on reddit. Trust me, even if you don't know anything about compression, it's well worth a read. This bright spark called Ashley Labowitz starts off by expounding on his fractal compression system that he's working on. He starts off by stating that the comp.compression FAQ is incorrect, which doesn't earn him many friends. He keeps going on about his system that he claims can compress any file. Unfortunately for him this is provably false.

Think about it this way, imagine I have a file that is two bytes long. Now, explain to me how that can be compressed to 1 byte or less. Simply put, it can't. There's no way, and anyone who thinks that this is possible has a very poor understanding of compression. Worse even than mine.

So, to cut a long story short, Ashley had completed his compression system, and, since he was running out of hard drive space, decided to use it to compress all his files, reasoning that he'd be able to access them when he completed his decompressor. Yep, that's right, he hadn't worked on or tested decompression yet!

So, basically he hasn't had any luck with the decompressor, much to the merriment of all and sundry.

Lessons learnt

1. When people in a newsgroup are calling you an idiot, it's possible that you are in fact being an idiot.
2. When people are telling you that something is impossible, and you're struggling to get it working, it's possible that they may be right.
3. FAQ's are there for a reason, and unless you're a recognised leader in the field, it's probably a good idea to take them at face value.
4. Saying "Before I become famous, I want to always remember the people that helped me get where I am" just before you make an abject embarrassment of yourself on the Internet is called hubris, which is, my father taught me, inevitably followed by nemesis.

How, we ask ourselves, with tears in our eyes, can somebody make such a huge mistake? Well, it's quite simple, and we're all guilty of it to some degree or another. We have a culture that tries to find value in every person, that makes an effort to save every life, that allows everyone to vote, and allows everyone to express an opinion. This is a good thing in general, however it can lead to this kind of arrogance. When no-one gainsays you enough, you start thinking that your opinions carry more weight than they actually do. You start ignoring opinions that don't agree with yours. Happily, reality tends to provide a swift correction. Unfortunately this sometimes means that you, for example, lose all your files.

Sometimes, especially when surrounded by people who believe the same incorrect things that you do, the fact that reality does not conform to your views is not readily apparent. Typical examples would be politicians, celebrities, politicians, creationists, and politicians. The decision making becomes poorer and poorer until eventually the decisions coming out of the affected group become wildly discordant from reality. One of the worst things you can ever do is surround your decision makers with sycophants. Nay-sayers in any organisation are absolutely critical in order to come to valid decisions.

Nitwits who by their actions try to show companies and the public that the GPL is run by a bunch of underpants-on-the-head kind of commie pinko weirdos.

Here's a hint Harald, and what the hell, let's include RMS in this too, most people involved in OSS or semi-OSS stuff are in it to help the community and not to push some radical political agenda. When you give them no choice but to sign up for your latest crusade, then quite a few will drop out.

Free as in do-what-I-tell-you-or-else free

After my previous bad experiences with the Operating System That Was Shipped in Beta, you might be surprised to find out that I recently installed it on my laptop. Interestingly enough it is far more stable and faster on my laptop than my desktop. My personal theory is that my laptop does not have an nVidia graphics card, nor a Creative sound card. Both are companies whose Vista support was late and uninspiring. Anyway, the number one, most irritating bug that I encounter regularly is file copy. Yep, you've heard me right, Vista cannot adequately perform operations on it's filesystem. I wanted to delete a directory holding backups, so I selected the folder, and pressed Shift-Del, and then had to wait for 2 hours whilst Vista carefully went and, from what I can tell, lovingly worked out the thumbnails for each file it was going to delete before actually doing the deletion. Oh, wait, it could also have been checking them for DRM.

Copying is so pathetically slow for anything large that in some cases, it's actually easier and faster and less frustrating for me to boot up a VPC containing Windows XP, share the appropriate folders on each Vista machine and use the XP to move the files from the one machine to the other.

Microsoft have been made well aware of this bug, but have not yet addressed them. In fact these bugs were raise during the Beta program IIRC, and still, Microsoft have not yet deigned to fix them. Copying files, deleting files..., high-tech shit.

I was inspired to blog this, because this morning I decided I'd like to watch a movie in bed. I powered up my laptop, and selected a movie from my desktop PC, and tried to copy it across. It started "Calculating time remaining", and had 0 bytes/second, and just sat there. I pressed Cancel, and the Cancel button just disabled. I waited and waited, and still that useless, buggy as all hell, copy dialog just did nada. So, I force restarted my laptop and tried again. When the exact same nothing happened, I decided to blog about it. So here I am, still staring at the 0 bytes/second, and still "Calculating time remaining". Here, Vista, a hint, if you're consistently at 0 bytes per second due to some unfathomable bug in your non-release-ready code then the time remaining is infinity.

Microsoft assured us that Vista was production ready, and it is actually a lot better than I feared it would be, but it is not release quality code. It appears that consumers are well aware of this, which is why Microsoft's financial results showed little boost from Vista sales. Microsoft have been engaging with consumers attempting to convince them to purchase Vista and not wait until SP1 comes out. This is a common tactic used by consumers to ensure tha most of the bugs are resolved. Such a tactic will mean far less frustrations for consumers, instead of acting as broad-scale beta testers, they will be hopefully actually be getting the code that should have shipped. I'd like to add that I don't think they should upgade existing PC's but only put Vista on new PC's due to the plethora of driver problems experienced by user of older machines.

Good Points

Is Vista worth the upgrade? So, far, just using my laptop as an example, yes. It's snappier, more responsive, much prettier. I find myself using the one-button off/suspend feature a lot and not bothering to properly shut down like I used to under XP, there's just no need. The search start menu feature has made my life a lot easier. Before, I'd go through my menus, setting up shortcut keys everywhere. Now there's no call for that. Mobility is also better, but I still wish they'd store IE settings like the proxy server against a named network, so I didn't have to keep fiddling that whenever I move from A to B. So, I am quite confident that Vista will be a huge success once they fix all the irritating little problems.

Speaking of which, I wish I could tell it to stop whining that I blocked some starup programs. I did it very deliberately, and it was a well-thought-out idea. Every 30 minutes or so, it pops up a bloody tooltip telling me that it did what I told it to do, and that it followed my instructions when I booted my computer 15 hours ago. I KNOW dammit!