Coding Sanity

There's been a lot of talk about the iPhone and how it exemplifies good design, yadda yadda yadda. However, what I'd like to touch on today is a piece of design which frankly left me convinced that every person on an entire product team should be fired. I have this feeling often, but in this case it's not a feeling, it's a fact. Every single person on the Microsoft Windows Mobile team should be fired immediately. No exceptions.

How can I possibly say that? Well, given the numerous bugs, inconsistencies, and general usability nightmares that is Windows Mobile I'd think it'd be pretty clear. However I'm not going to address that. I'm not going to address companies like HTC whose, should we say, unique ideas about how to implement a keyboard leave me taking 4 times as long to write an SMS with my QWERTY keyboard as my girlfriend with her 1234 keyboard. I'm similarly not going to scoff about the idiocy that is the Mobile contact system, or even worse the stupid inconsistencies between how you address an MMS as compared to an SMS. And if that blasted dictionary suggests ONE more hugely long word when the far more common short word fails to appear in it's choices...*sob*

 ...

....

Okay, I'm back. I've dried my eyes and am now prepared to recount one of the most STUPID WTF's in programming history!

Imagine some friends at a table having dinner. Imagine that one of the friends, being a complete moron, has actually been so mindbogglingly stupid as to have shelled out actual cash for the POS that is a Microsoft Windows Mobile HTC POC*1000 (Piece of Crap, new model). Let us imagine that these friends have an argument, let us further imagine that the Stupid Moronic Twit decides to settle the argument by opening the relevant Wikipedia page on his POC1000. Let us now skip past the imaginary scrolling down and down and down because the POC1000 renders wikipedia as a 5-char wide web page.

Let us now arrive at the delightful point when he starts reading the portion that proves him right: "...to its diameter in Euclidean space... huh! WTF!?!" as the phone brings up, for a split second a dialog reading something along the lines of "The storage system is short of space, please clear some space immediately".

Being a well trained monkey, used to the insane vagiaries of the POC1000 (e.g. the random requirement to align the screen before continuing, no matter that you were trying to phone the ambulance service or police) he immediately enters Settings, opens the Clear Storage application, ... and... the phone reboots. *Sigh* ah well, that's what you get for choosing a company that can't write software to save their lives, isn't it?

Not so fast. Gather round to feel the true horror...

"Wait, why is it telling me that it's setting itself up for the first time?"

"Why is it making me answer stupid Customer Improvement queries?"

The reality of course being that we don't need better customers, we need a better f****ing supplier.

"Please, please, oh dear sweet Lord, please tell me that it hasn't wiped everything on my phone because it was marginally short of space for 5 seconds?!?!?"

Surely any halfway competant engineer would know to delete the temp files and temporary internet files before moving on to contacts, emails, SMS's and system settings? Ah, there you see the problem. We're assuming that the numbnuts who coded this nighmarish piece of fly strewn horse manure were in fact competant.

So here's my question for the day: "When is it acceptable to wipe your user's data in order to ensure that your stupid program can keep on wasting resources?"

An extra 10 points if you work on the Microsoft Windows Mobile team.

This is basically just a quick response to Roger's "WTF on WTF" rant since he doesn't allow comments. Basically, my advice boils down to: get over it. Someone, somewhere will always call WTF your code. The first thing to remember is that it's your code the person is critiquing not you. The second thing to keep in mind is that you can use such criticism to become a better developer. What doesn't work (trust me on this) is becoming defensive and angry.

"What makes a person so arrogant to mock somebody elses error?"
Skill, experience, and yes a fair bit of arrogance. All the best developers tend to be a bit arrogant. Keep in mind the reason Pieter was writing his WTF articles: to educate developers on what to do, and what to avoid. He could have done this with a lengthy article about the pros and cons (like I would write), but instead he chose to keep them short and to the point. The easiest way to do this is to just say "A is bad, B is good", or just "A is bad". Simple, easy little "rules" to help all the other developers stay on the track.

"Not everybody is at the same level in programming"
Absolutely, which is why articles that point out common mistakes that people make are good. By writing helpful articles, bad habits in the readers are hopefully reduced, and such developers become better. You only improve via learning, and the only way you'll learn is if you pay attention to teachers.

"Some are experts in one language but when migrating to another language you might be used to certain methodologies"
Indeed, but again, how are you going to learn that these mistakes are bad except by paying attention to the guys who are good in your chosen new language? In any case, I can't find any recent posts that point out something that is bad in C# but good in another language. All of the items give pretty good advice which would apply to any relevant language. I have my own minor disagreements with one or two of Pieter's posts, but on the whole, I'd still recommend them without a moment's hesitation.

"There are many reasons why bad code slips in"
Absolutely, 100% agreed. Roger goes on to mention legacy code, which is a common, but by no means exhaustive source of bad code. In my experience, the number one cause of bad code is developers who refuse to listen to criticism. The moment you mistake a critique of your code with a critique of you you get defensive, shut down, and get upset. This ensures that you do not learn from your mistakes. Your code isn't you, and frankly you should be grateful that people are picking up your mistakes and letting you know about them.

Conclusion
Roger then goes on to say

"Writing code in a programming language is like writing a document in the English language. Would you post a blog if you found a grammatical error in somebody else's written document? Or if a foreigner gets their grammer mixed up do you laught at them? No, its stupid and a waste of time."
Umm, do this search "gramattical errors blog" and you'll find almost 3 million results. So, yes, people do critique gramattical errors, for the same reason we critique code, to hold up bad examples and tell people "don't do it that way, this way is much better"

So, a plea to all developers who have seen some code they've written wind up in a WTF-type article, use the experience positively, don't get defensive about it, don't get angry. Use it as a learning point. Maybe that mistake is not the only one, maybe it's just a corner of a whole raft of mistakes you're making. A little research could massively improve the quality of your code. Even better, maybe there was a very good reason for what you did. You can then explain why you did it that way in order to teach that there is not always an absolute right and wrong, that sometimes there are exceptions. In a sense, that's pretty much what Raymond Chen's blog The Old New Thing is about, explaining the reasoning behind things in Windows that seem silly or odd.

The whole "Is LINQ to SQL dead or not" thing is making me angry and irritated all at the same time. However, I've decided that instead of complaining I should rather offer a public service. So, what I'm going to try and do is try and teach Tim Mallalieu, the Program Manager for LINQ to SQL and LINQ to Entities some straightforward English. He's written some clear posts before, notably his defence of Entity Framework against the Vote of No Confidence in it, so he's certainly capable of writing without vagueness. However, his two posts about the future of LINQ to SQL are pretty vague and covered in weaselese, so I'm going to try and help him regain his clarity.

Let's start with Update on LINQ to SQL and LINQ to Entities Roadmap:

Since the release of LINQ to SQL and the Entity Framework, many questions have been raised about the future plans for the technologies and how they will relate to each other long term.
During this week of PDC we are now at a point, with the announcement of Visual Studio 10 and the .NET Framework 4.0, that we can provide more clarity on our direction.
We have seen great momentum with LINQ in the last year.  In .NET Framework 3.5 we released several LINQ providers, including LINQ to SQL which set the bar for a great programming model with LINQ over relational databases.  In .NET 3.5 SP1, we followed up that investment with the Entity Framework enabling developers to build more advanced scenarios and to use LINQ against any database including SQL Server, Oracle, DB2, MySQL, etc.
We’re making significant investments in the Entity Framework such that as of .NET 4.0 the Entity Framework will be our recommended data access solution for LINQ to relational scenarios.  We are listening to customers regarding LINQ to SQL and will continue to evolve the product based on feedback we receive from the community as well.

Tim Mallalieu
Program Manager, LINQ to SQL and Entity Framework

Okay, there's a whole lot of filler here, so let's see if we can trim it down for him:

I'm going to tell you about the future direction of LINQ to SQL and LINQ to Entities. LINQ to SQL was unaccountably popular after it's release and has remained popular even though we released Entity Framework. We've spent a lot of effort making Entity Framework more compelling, but it doesn't seem to have helped it's popularity enough.

So, we've decided to just do bug fixes on LINQ to SQL from now on and concentrate all our efforts on Entity Framework.

He followed this post up with a real gem "Clarifying the message on L2S Futures.", let's translate it piece by piece:

There has been a variety of responses to the post on L2S futures in the last couple of days.

Let me start by saying sorry for the radio silence since the post but, as Elisa mentioned,  we posted it while at PDC and were focusing on the interactions there over the last couple of days.

There are a couple specific points that I would like to clarify:

This is not too bad I must admit, but let's look at that phrase "a variety of responses". Hmm, if I look at the comments on the previous post, I get the following:

  • 32 pro LINQ to SQL or anti Entity Framework
  • 6 pro Entity Framework
  • 5 mixed
  • 14 trackbacks or other types of comments

Admittedly, I gave up counting about halfway through the comments out of sheer boredom, so there could be an uncounted late surge for Entity Framework, but I'm going to call this one. Anyway, if we look only at the comments related to this issue we get:

Graph

Hmm, I admit that this could be called a "variety of responses", but my money would be on "overwhelmingly negative responses".

Anyway, let's continue:

Is LINQ dead?

No… heck no…

There is a big difference between LINQ to SQL and LINQ.

LINQ is Language Integrated Query, today we ship the following sources over which one can execute a LINQ query:

  • DataSet : LINQ to DataSet
  • XML: LINQ to XML
  • In Memory Objects: LINQ to Objects
  • ADO.NET Data Services (Astoria) Client: LINQ to Data Services
  • LINQ to Relational Databases:
    • LINQ to SQL – the technology we were discussing in the original post
    • Entity Framework (LINQ to Entities)

There are many other people in the company and broader community working on LINQ solutions to other data sources. We are also excited to see LINQ being applied to many cool new problems beyond the typical data access scenario.

The discussion in the post was about the difference in investment level that we will have going forward regarding LINQ to SQL and the Entity Framework. LINQ itself, is a fundamental technology that we will continue to bet heavily on.

Okay, I'm not sure how many people thought that LINQ itself was being killed, but I'm guessing not many, so this portion is a clear strawman that serves no useful purpose, and we could cut it.

So what exactly is the announcement about?

Over the last few months we have been looking at how to carry forward LINQ to SQL and LINQ to Entities. At first glance one may assert that they are differentiated technologies and can be evolved separately. The problem is that the intersection of capabilities is already quite large and the asks from users of each technology takes the products on a rapid feature convergence path. For example, common asks for LINQ to Entities (that are being delivered with .NET 4.0) are POCO and Lazy Load.  Similarly, in the LINQ to SQL side we are being asked to provide new mapping strategies and other features which the EF already has. Additionally there are common asks for new features like UDT’s and better N-Tier support for both stacks. The announcement really centers around the point that,  after looking hard  at this and triangulating with internal partners and customers, we decided to take the EF forward with regards to the overall convergence effort and over time provide a single solution that can address the various asks.

Buzzphrase list: "carry forward", "differentiated", "evolved", "intersection", "asks", "convergence", "strategies", "triangulating", "solution"

Right, let's try this again

So what exactly is the announcement about?

Over the last few months we've been thinking about the future of LINQ to SQL and LINQ to Entities. Initially they may seem to be different technologies, but there are significant overlaps. Many feature requests for the one technology already exist in the other, and many others are common to both. The announcement basically says that we decided to settle on a single solution, and that would be Entity Framework.

Next part:

Is LINQ to SQL Dead?

We will continue make some investments in LINQ to SQL based on customer feedback. This post was about making our intentions for future innovation clear and to call out the fact that as of .NET 4.0, LINQ to Entities will be the recommended data access solution for LINQ to relational scenarios. As mentioned, we have been working on this decision for the last few months. When we made the decision, we felt that it was important to immediately to let the community know. We knew that being open about this would result in a lot of feedback from the community, but it was important to be transparent about what we are doing as early as possible.  We want to get this information out to developers so that you know where we’re headed and can factor that in when you’re deciding how to build future applications on .NET.  We also want to get your feedback on the key experiences in LINQ to SQL that we need to add in to LINQ to Entities in order to enable the same simple scenarios that brought you to use LINQ to SQL in the first place.

Hmm, what are "investments...based on customer feedback"? I also love all this talk about being open and transparent whilst using weaselese in abundance. Here's the translation:

Is LINQ to SQL Dead?

Yes, but we will fix bugs.

Tim Mallalieu
Program Manager, Entity Framework

Conclusion

Personally, I'm kind of conflicted about this. On the one hand I see their point that there should only be one data access strategy, and LINQ to Entities is clearly the bigger brother in this regard. If it were any other people running this, I'd be cheering from the sidelines. However the Entity Framework team have a history of ignoring their customers, being opaque and secretive, and delivering a bloated and inefficient product that did not address the most common requirements, despite being warned that this would happen. They've promised to be more transparent and open, but these two posts are opaque and vague.

Personally, I wish them luck, and hope that they can indeed deliver a system that merges the best of LINQ to SQL and LINQ to Entities.

However, until such time as they actually start speaking English and being forthright, I won't believe them.

Right, so in Part 1 we looked at the requirements for a Discovery Service and in Part 2 we looked at how we'd host the WCF service. Now it's time to look at the UDP itself. First off though, if you look in Part 2, at the declaration I've got for DiscoveryServer, you'll see I'm inheriting from a DiscoveryBase. This base class will handle most of the UDP stuff, since both DiscoveryClient and DiscoveryServer need a very similar set of operations.

Let's look at the most important method on DiscoveryBase:

private UdpClient _udpClient;
private long _connected;
private int _timeToLive = 2;
private IPAddress _address;
internal void Prepare(IPAddress address, int localPort)
{
    // Ensure we're not already discovering
    if (Interlocked.CompareExchange(ref _connected, 1, 0) == 1)
        throw new InvalidOperationException("Already in discovery mode.");

    // Create the client
    _udpClient = new UdpClient(localPort);

    // If it's broadcast we set the broadcast flag
    if (IsBroadcast(address))
        _udpClient.EnableBroadcast = true;

    // If it's multicast we join the multicast group
    if (IsMulticast(address))
        _udpClient.JoinMulticastGroup(address, _timeToLive);

    // Store the location to connect to
    _address = address;
}

As you can see, it's not terribly complicated. There's a little bit of concurrency logic to ensure we're not already discovering, we create a UdpClient object, and then either set it to broadcast or join a multicast group depending on the IP address. Finally we cache the address so that our child classes can access it via the Address property. _timeToLive is an interesting property, it defines how many router hops any multicast broadcasts will go. For now, I've just set this to 2, but you could easily make it configurable.

Now let's hop back to DiscoveryServer and have a look at how it kicks off it's discovery:

public void Publish(IPAddress address, int port)
{
    Prepare(address, port);

    // Set up the discovery service
    _host = new ServiceHost<IDiscoveryService>(this, _address);
    Binding binding = BindingAddressParser.CreateTransportBinding(_address);
    _host.AddServiceEndpoint(typeof(IDiscoveryService), binding, _address);
    _host.Open();

    // Now asynchronously listen for any incoming requests
    Client.BeginReceive(Receive, null);
}

So, as you can see, it calls Prepare on DiscoveryBase, in order to ready itself for UDP operations. It then begins hosting the IDiscoveryService on a transport and binding determined by an address which is passed into the DiscoveryServers constructor. Finally it kicks off an asynchronous receive operation on the UDP client. Let's have a closer look at Receive:

private void Receive(IAsyncResult ar)
{
    if (Connected) // This will be false if the UDP client has been closed
    {
        IPEndPoint remoteEP = null;
        byte[] result = Client.EndReceive(ar, ref remoteEP);

        // Now asynchronously listen for any incoming requests
        Client.BeginReceive(Receive, null);

        // Is there actual information to fetch?
        if ((result != null) && (remoteEP != null) && (result.Length > 0))
        {
            string message = Encoding.ASCII.GetString(result);

            if (message == @"get\services")
            {
                // Handle the get services request by sending the address where the service information is hosted
                message = BindingAddressParser.ResolveAddressHostName(_address);
                byte[] response = Encoding.ASCII.GetBytes(message);
                Client.Send(response, response.Length, remoteEP);
            }
        }
    }
}

So, what happens is we check if we're connected (the Connected property on DiscoveryBase returns whether the _connected field we saw in Prepare is equal to 1), we then kick off another asynchronous receive just in case there's more requests on the way. Then we decide whether it's a valid request. If it it, we send the address where we hosted the IDiscoveryService.

In Part 4 we'll start looking at the DiscoveryClient.

Right, so in Part 1 we looked at an overview of how this discovery system should work. Now let's start looking at specifics. Before we start diving into the UDP stuff, let's look at the information that the server will publish and how this will be accomplished. If you recall, the mechanism I suggested was that the server host a normal WCF service that held the information about the services, and that the UDP would be used to provide access to this service. So let's start with the WCF side.

First off we're going to require our ServiceInfo class. This is a relatively simple class, which contains an Address property and a set of Name/Value pairs. Needless to say, it needs to be decorated with the DataContract attribute. To provide access to this information we will have the IDiscoveryService interface:

namespace Sanity.Net
{
    [ServiceContract]
    internal interface IDiscoveryService
   
{

        [OperationContract]
        ServiceInfoCollection GetServices();

   
}
}

You'll note that I made this interface internal, since there's no real call for outside applications to access it without going through the Discovery system we're writing. Our next step is to create a DiscoveryServer class which will take the address it needs to publish as a parameter in its constructor. In order to keep things simple, I'm only going to allow the default binding for the transport mechanism. As I discussed in Part 1, discovery is not the place to get fancy with communication methods, it's where you find out about the fancy communication methods.

We'll create a Publish method on the DsicoveryServer, that will create the service host and open the service endpoint:

private string _address;
private ServiceHost<IDiscoveryService> _host;
public void Publish()
{
// Set up the discovery service
_host = new ServiceHost<IDiscoveryService>(this, _address);
Binding binding = BindingAddressParser.CreateTransportBinding(_address);
_host.AddServiceEndpoint(typeof(IDiscoveryService), binding, _address);
_host.Open();
}

Now, you'll notice my BindingAddressParser, that's just a little class that will infer the binding from the address, NetTcpBinding for "net.tcp" and so forth.

Our next step is to add our ServiceInfoCollection to the class so that services can be given to the discovery server:

private ServiceInfoCollection _localServices;
public ServiceInfo AddService(string address)
{
address = BindingAddressParser.ResolveAddressHostName(address);
return _localServices.Add(address);
}
Now an interesting item is that ResolveAddressName. If you think about it publishing an address like "net.tcp://localhost:21021/MyService" is not terribly useful, since we need to know where localhost actually is. So ResolveAddressHostName uses DNS to replace localhost with the machines actual name. It will also resolve an IP addresses to the machines name as well (e.g. 127.0.0.1).

Finally, we can implement the IDiscoveryService interface:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]
public class DiscoveryServer : DiscoveryBase, IDiscoveryService
{

ServiceInfoCollection IDiscoveryService.GetServices()
{
return _localServices;
}
}

Okay, we now have a server-side that will take services with arbitrary information and make them available on TCP/IP. In Part 3 we will look at making this available on UDP as well.

Just in case you thought secretly installing malware on a machine wasn't bad enough, it appears that Apple aren't actually interested in resolving security flaws in Safari. So it's bug-ridden and they're happy with that.

This has led Microsoft to take the unprecedented step of advising Windows users to restrict their use of Safari.

So, I guess the question of whether Apples products were 'more secure' than Microsofts due to inherent security or installed base has been definitively answered.

Metadata Exchange mechanisms such as WSDL are all very well and good, but they fulfil one part of the automatic discovery experience. If you look at WCF a service consists of three parts:

  1. Contract
  2. Address
  3. Binding

So, Metadata Exchange helps you figure out the Contract if you're given the Address and sometimes the Binding. All very well and good for assisting us developers get running in using a service. However, how about a scenario where you already know the Contract but you need to figure out the Address and/or Binding? In other words, what about most deployment scenarios? How can you discover where to find the server? One way would be to have a fixed location which your client connects to in order to query it for server information. However, if you can hard-config this repository, why can't you just hard-config the service location too? Setting the service location in the client configuration file seems to be a fairly common approach.

Another option would be to capture the service location during installation or when the application first runs. This is fairly common as well. However, this assumes a level of technical ability and knowledge on the part of the installer/user that may not be justified. Also, what happens if the service location changes?

UDP Discovery

So, what we would ideally need is a mechanism to help enable the various discovery options. Happily there is a technology ideally suited for discovery, and it's called UDP. UDP is a networking technology fairly similar to TCP/IP. Unlike TCP/IP it does not support connections, each message is sent fire-and-forget. There's no guarantee of delivery, and if your message is broken into multiple packets there's no guarantee what order they'd appear in. This makes UDP a bit hit and miss for most business applications, which is why it's very rarely used by most developers.

However there are two really cool features of UDP: broadcast and multicast. Broadcast allows you to do is to send a message to all the machines on the same subnet, which is fine for discovering the machines near you, but not so good to find servers, which are usually on a different subnet. Broadcast also has the downside that it sends a lot of traffic, one message to each machine on the subnet. Multicast is different. Unlike with broadcast, you have to join a multicast group, and then messages are routed to the machines in the group. Since this is only the machines interested in the message, the number of messages flowing across the network is smaller.

So, if we're looking for machines on our network, UDP would seem to be an ideal mechanism. We'd want to keep the messages small so as to avoid any issues with packet ordering. Now, if we're publishing generic information about various services, we don't really know how much information we'd be sharing. So ideally, we'd want to avoid sharing this generic service information across UDP.

What we can easily share is the location of a more reliable service from which this service information can be gleaned.

So the next question is how the whole UDP thing should work. A common approach is to have the servers broadcast their information at a fixed interval. This has the advantage of simplicity, but it isn't terribly elegant. Even ignoring the fact that the servers are often sending out data that no-one wants, the client machine would sometimes have to wait a while until the server next send out it's data.

So, whilst a bit more complex to code, it makes a lot of sense for the client to send out a request, and the server to respond.

Review

So, what we're looking at here is a three-phase discovery mechanism:

  1. Client sends out UDP request
  2. Server responds with address to discovery location
  3. Client connects to discovery location and retrieves data

So, the client UDP request would be fairly simple, some kind of "Please give me service information" message such as "get\services". The server response would also be relatively simple, an address e.g. "net.tcp://myserver:20231/Discovery". I'm going to assume that we're not going to get into complex binding scenarios such as security and transaction information. This is simply a location to get more information, so we can keep it to the default bindings.

The next question to ask is what information the server should share about the services it exposes? Well, the address of each service would be an obvious starting point. However, what else? Binding information? But that's WCF specific, and who said the services being exposed are WCF? So it probably makes sense to simply allow each service to have a set of properties: name/value pairs that contain contextual information about the service.

In Part Two we'll start looking at how we're going to implement all this.

Marcus Ranum has an article titled  The Six Dumbest Ideas in Computer Security. I strongly suggest that anyone interested in security read it, and his other articles too. It's quite an old article, but someone put it on proggit and it made me remember it. In fact, I'd say that this article was seminal in guiding my undertstanding of computer security. For those of you too lazy to click on links, a swift summary:

  1. Permitting things by default instead of denying things by default. Used to be very common in firewalls.
  2. Focusing on stopping bad things rather than only allowing known good things, a special case of #1, but still very common in anti-viruses and operating systems.
  3. Catch holes by fixing hacks as they become known rather than hardening the application.
  4. Hacking is cool.
  5. Educating users.
  6. Doing something is always better than not doing something. Think about it this way, do you really want to be running bleeding edge software for your firewall?

A point he makes about #3 and #5 is that if these worked they would have worked by now. IE would be the most secure browser on the planet and users wouldn't click on attachments anymore.

Do yourself a favour, go and read the original article. It's well worth it. 

I recently received a chain email from a friend where the original author was complaining bitterly about the massive electricity price hikes being mooted by Eskom (our state-owned electricity company) in order to pay for their build programme. The author felt that since it was Eskom and government who were asleep at the switch and allowed the power situation to reach its current dire state, that the electricity price should not be raised. Instead, the author suggested that the power stations be built using the surplus that government has accumulated over the past few years.

It's in a slightly histrionic tone, but I think many people would sympathize with the author's points, certainly the 278 people who had signed it by the time it got to me did. The Eskom situation easily inflames people since we're not only being slammed with a price hike, but also rolling blackouts, and to top it all off, a large amount of the problem seems to be from fat cats keeping their eye on profitability rather than on long-term viability. To top it all off, the stories about Eskom supplying Zimbabwe with free electricity added the outrage over our government's dithering on that country to the issue. What I'd like to do is look at things from a less emotional point of view. I'm incredibly lucky in having some inside information about this, since certain colleagues have done modeling on this issue. I don't unfortunately have the raw figures, so I'm going to have to try and remember them as best I can.

This is the situation we find ourselves in

The first thing to remember is that complaining about Eskom or government or whoever may make us feel a lot better, but it won't actually solve anything. What has to be dealt with is the situation that exists right now. What this boils down to is reserve margin. From the Wikipedia page about Eskom I see that their overall capacity is about 39,046 MWe. So, that's the amount they could produce in a perfect world. However, the world is not perfect, and there are situations where you have to take a power station offline in order to maintain it, and of course you have unplanned outages as well. So, the idea is that the total amount you can produce exceeds what you need by a margin, the reserve margin. It is this that has been allowed to drop too low. Eskom have enough generating capacity, in theory, to supply all of South Africa's needs. However, to do that, they'd have to stop maintaining the power stations, and we can all guess where that would lead us. The ideal reserve margin seems to be at about 15%, and ours is about 6-8% right now.

Now an important point is that this reserve margin is over our peak usage. The usage of electricity during the day is similar to the graph you see here. Electricity usage is low at night, climbs rapidly in the morning, dips a bit during the course of the day, and then climbs in the evening before settling down again.

So our problem is that our usage is rising above the ideal reserve capacity during the day, and this is why Eskom is being forced to have these blackouts (I refuse to call it load shedding, its bloody rolling blackouts, not some saccharine platitude).

So, what can we do about this situation? Well, two main approaches are fairly obvious:

1. Increase supply

2. Decrease demand

Let's look at them in a bit more detail.

Increase Supply

Well, this is nice and simple it would seem: build more power stations. All well and done, but it costs money to build power stations. More importantly, the big ones take a long time to build. The following are some of the things we can do:

Buy more power from neighboring countries

Unfortunately we're by far the biggest producer and consumer of electricity in the region, so the amount of power we can buy is limited. Eskom recently upped the amount we get from the Cahora Bassa hydroelectric station by 250MWe if I remember correctly. It will help, but it's a drop in the ocean.

Build more power stations

There are a few main kinds of power stations we can build. Coal powered stations are huge and take a long time to build. On the upside they produce vast amounts of power very cheaply. Eskom are building two huge coal powered stations called Bravo and Medupi which will start coming online in about 2012. They are also de-mothballing another three, but they're not nearly as big. Another main type of station they can build is gas turbine. These take far less time to build, but they also produce far less. Unfortunately they also cost vastly more to run as well, so the electricity from them is very expensive. Normally, because they can be switched on and shut down fairly easily, they are used only to deal with demand spikes. Hydroelectric has the advantage of being fairly green and fairly cheap. Unfortunately we live in an arid country, so the amount of these we can build is limited. Finally, we have nuclear stations. These make an awful lot of sense, the electricity from them is fairly cheap, like coal, they're "base load" stations. But, like coal stations, they take a long time to build and are very very expensive.

"Green" electricity production such as wind farms and solar is also a possibility, but tends to be fairly limited in the amount of power they can produce as well. Plus, they do tend to be expensive. One of the nice things about them is that they can start coming online fairly swiftly, and every little bit helps.

So, it looks like a good strategy would be to build some green producers and gas turbines, and boost the amount of coal and nuclear stations in the long term. This is in fact exactly what Eskom are proposing to do. Again, I might have these figures a bit off, but they plan on spending about R1, 3 trillion over the next 10-15 years on building new capacity. This is an awful lot of money, and it has to come from somewhere.

Encourage cogeneration

Cogeneration is quite interesting. There are quite a few industries in South Africa that produce an enormous amount of heat from their processes. What happens with cogeneration is that this heat is channeled into turbines which then provide electricity to the national grid. Now, a company is not going to do this out of the goodness of their hearts, they're going to expect to be paid for the electricity they supply. Unfortunately, the low price it is at right now is not enough to make this feasible. However, the increases Eskom is proposing will move us into a position where cogeneration will become viable. The amount of cogeneration capacity may be as high as 5,000 MWe, basically the same as a huge coal-fired power station.

Encourage independent power producers

This is in fact what government was hoping for, why they didn't want Eskom to build more power stations. They were hoping that companies would come to South Africa and build power stations. Unfortunately, for some reason, companies did not flock to compete in an environment where their only customer set the price, and more importantly the price was too low for anyone to compete with. Building a power station is expensive, and you have to recoup your capital costs somehow. Eskom didn't have to, since they already had the power stations, and had pretty much paid them all off. Anyone else coming onto the market would have to compete in an environment where they would be unlikely to be able to make a profit. Interestingly enough, the price increases that Eskom is proposing will likely make potential independent power producers interested. It would now be possible to compete. This is largely because Eskom is now undergoing the same kinds of massive capital costs an independent producer would have to go through.

Encourage energy-negative houses

In some countries, the power utilities are required to buy back any power you produce at home. So, if you have solar panels and your house is producing more power than it consumes, then you get paid back. This is a quite nice idea for encouraging energy efficiency, but it is unlikely to be terribly effective in South Africa. Even with Eskom's price increases, electricity would still not be so expensive as to make this profitable for the individual concerned. In any case, it is feasible for only relatively well off households, which is a small percentage of the total.

Decrease Demand

This one is fairly obvious, since Eskom are using rolling blackouts as a mechanism to force decreased demand. What are some other alternatives?

Encourage energy efficiency in households and offices

This is fairly obvious. Switching to energy efficient light bulbs can produce a marked saving in power, geyser blankets and timers similarly help. More importantly, timers can be used to drop demand during times of peak usage. If every single person who has a geyser switched it off from 6am to 7pm, they wouldn't greatly inconvenience themselves and it would make a significant reduction to peak load. Switching from electricity to gas would probably help a bit, but not as much as you'd think, largely because you don't really cook during peak times. Tumble dryers and air conditioners are proverbially profligate in their energy usage. You could look at solar water geysers, but I think you'll find that a timer is just as effective, and much cheaper.

Apparently up to 15% of your house's energy usage is consumed by appliances which are plugged in, but "off" in a standby mode. Your TV, hi-fi, cell phone charger you left plugged in and so forth. We could encourage people to unplug these.

However, this raises a problem. All these ideas either cost the consumer money or they are a pain in the neck. Why would people do these things? South Africa is remarkably wasteful when it comes to electricity, a hangover from decades of cheap power. I've heard numerous suggestions such as punishing people who use "too much" with punitive fines. However, who decides what's too much? What if I have 6 people in my house and you have one in yours? It becomes very, very tricky. However, capitalism has a nice and simple way of encouraging people to use less of a thing whilst at the same time punishing those who use "too much". It's called raising prices.

Unfortunately, people can easily become habituated to small changes, so to force people's behavior to fundamentally shift you have to shock them, and a big increase is likely to accomplish that, whilst numerous small ones will probably not.

Shut down industries

This is not a good one. Shutting down mines and aluminum smelters saves power, but it harms the economy as well. Badly. In any case, as with normal consumers, the best way to get industry to use less power is to charge them more for it. A lot more. This will lead to inflation, true, but it is far better to see some inflation rather than entire sectors of our economy being brought to a standstill. Unfortunately our Reserve Bank has an itchy trigger finger when it comes to inflation and is pursuing a punitive interest rate policy to keep it in check. I think we will have to let the Bank allow the inflation caused by Eskom's increases to work their way through the economy without raising interest rates, even if it does mean some spiking in inflation. If not, the Bank may all but shut down the economy. This would help conserve power, but not in a way any of us are likely to enjoy.

Punitive blackouts

This is what we have right now, where rolling blackouts happens no matter what, unless you cut your usage by 10%. The only problem is that Eskom and the municipalities are not geared to blackout small areas, so it's groups of suburbs that go down together. Ideally, in my opinion, they should try and get to a point where a blackout unit is about 10-20 houses. Then it becomes feasible for communities to encourage and work towards energy efficiency in their area. If they do so, and cut their usage by 10%, they stop getting blacked out. Right now, it's not really practical to lead a community effort to cut usage by 10% across all of Johannesburg, but a street, now that's doable. When you're sitting in the dark and the guys in the next street have lights, maybe you'd consider changing your light bulbs or putting in other energy saving mechanisms.

Denying access to power

This is another approach that Eskom are using; they're simply refusing to provide power to new developments. The long term effects of such a strategy will be horrific though. What they're essentially doing is stopping most new fixed investment in South Africa, investment we critically need for our economy to grow. This cannot continue for very long without causing a massive problem.

Changing the peak structure

This is one I find personally interesting. The idea here is not so much to conserve power, but rather to shift it, to flatten out the peaks of demand by moving that demand to the troughs. An obvious way of trying this is to charge more for peak times, just as the cell phone operators do, maybe even give "free" power away in off-peak times. Again, the difference would have to be big enough to make people sit up and take notice.

Another very interesting proposal has to do with time zones. The idea here is that we split South Africa into two time zones. I'd guess that Cape Town, Port Elizabeth and Bloemfontein would fall into the one time zone, and Johannesburg and Durban into the other. This morning the sun rose at 07:11 in Cape Town and at 06:17 in Durban. So, if South Africa is in two time zones, an hour apart, that usage pattern above would be flattened out a bit, and widened. I personally find this idea fascinating and well worth the attempt. Changing time zones would not cause too much inconvenience in any case. For goodness sakes, have you ever tried to get hold of a Capetonian before 9am? Not a chance. So, this would just formalize that Smile

Going forward

Quite simply, the price hikes that are being proposed are important and necessary. There is the worry about how they will affect the poor, obviously, and how the inflation created would similarly add pressure to them. However, the poor would be worse affected by our economy tanking and their ranks being swollen with newly unemployed. Right now, the fundamentals of our economy are in pretty good shape. Besides our trade deficit which is a bit worrying, the only other cloud is this power problem. If we solve it, then there's no reason our economy can't carry on growing. Of course, that would increase demand for electricity even more. You win some and lose some.

Anyway, I digress, the point is that there is a lot we can do to increase supply and most of it is being done or planned for. However, to increase supply enough will require a significant increase in electricity prices; effectively they are going to have to double over the next two years. The problem is that the supply increases will largely take a long time to come on stream; we simply can't build the power stations quickly enough. Therefore, for the short and medium term we have to look at constraining demand. Blackouts and stopping new developments will quite simply bring our economy to a screeching halt. We're doing these now, because we don't have any choice, but we must aim at getting rid of these measures at the earliest possible time. That means consumers, business and industry coming to the party on saving power. Eskom and government's pleas to consumers and business have not had much effect on people's behavior.

Personally, I think part of the reason for that is the anger at them for letting this situation develop in the first place, there's a sense that it's "their problem". Unfortunately, it's not, it's our problem, and will remain so for some time to come. A lot of people are understandably unhappy about the proposed price increases and see it as being punished for someone else's mistakes. Certainly there is an element of that, but think about it this way. For the last decade or so, you've had electricity at a discount. Those discounts are coming to an end. Sure, it's not nice, and it would be easier to swallow if the lights stayed on the whole time, but they were at a discount, and prices are just moving to where they should have been. Eskom and governments mismanagement extended to not increasing prices as much and as fast as they should have been. So in a sense we all benefited from their mistake in the past, and are suffering for it in the present.

A final thought. Whenever I have issues that afflict me, work or personal, I always try and think about two things:

  • What could I have done differently?
  • What can I do in the future?

It very rarely helps matters to worry about who to blame, all that does is distract you from the solutions. In this case, yes, there are definitely people to blame, and I doubt there's much many of us could have done in the past, since we didn't even know about this problem. However, we can look to the future. By conserving power, especially at peak times, even if all you do is cook a little later or install a geyser timer you can actually make an important difference. More importantly, by just being a little more energy conscious you will be planning for your future.

I assure you, this power issue is the tip of the iceberg. Energy costs are rising worldwide, and this trend is going to continue. We will look back on this power cost increase in a decade and laugh at how little power cost in 2010.

There seems to be a serious issue with Microsoft lately, and it seems to be a strategic and executive issue. They have, for a couple of years now, been desperately running around trying to own every technology market they're not in whilst at the same time allowing Windows, their core product to "collapse". Witness their doomed attempt to take on the iPod with the Zune. The question has to be asked, why? I mean, I can understand why Creative or Sony or another portable media company might try and rush out ill-advised competitors to the iPod, it's a major part of their business after all. But why for goodness sakes would Microsoft decide to spend a fortune taking on an established incumbent? Especially after waiting years for them to cement their lead before getting involved?

Forget the numerous complaints about the Zune itself; I'm just trying to understand their strategy here.

Let's not forget their decision, many years ago, to take on the games consoles. Well, that seems to have worked out okay, although it sure cost them a ton of money. Now, why did they do that? Well, the worry was that consoles might supplant PC's, a reasonable concern I guess, unlikely, but possible. They sure couldn't have done it to sell software, since MS games were few and far between at the time.

So now we get onto their biggest strategic blunder; their desperate attempts to unseat Google as the king of search. It seems that Ballmer and Co are so terrified of Google and the *gasp* massive advertising revenues they earn that they are literally willing to bet the entire company on a chance, not a certainty, but a chance that they can eat into this revenue stream.

Financials 

Let's look at some figures, to be precise the 2007 Q4 earnings reports from Microsoft and Google:

Market Cap
Microsoft Google Difference
Total Assets $63,171,000,000 $25,335,806,000 $37,835,194,000
Net Income $3,035,000,000 $1,030,716,000 $2,004,284,000
$266,180,000,000 $145,440,000,000 $120,740,000,000

So, Microsoft has assets of 2.5 times Google's, it's income is 3 times, and it's market cap is 1.8 times. Now, this is by no means a complete picture. For starters it is my opinion that Google is in fact significantly overvalued, however I'm certainly no expert. Up until fairly recently I'd also have said that Microsoft is undervalued, but I no longer think that. This is not because I think that the company as a whole is valued correctly, I just feel that the utterly idiotic actions recently demand a discount to be placed onto the Microsoft valuation.

Yahoo!

To be precise, the idiocies to which I am referring is the Microsoft desire to buy Yahoo come hell or high water. As a matter of interest, Yahoo has a market cap of a mere $37,860,000,000, meaning that a "perfect" merger that inflated Microsoft's market cap by Yahoo's would result in Microsoft gaining a mere 14%. Needless to say, no merger goes that easily. In this case you're merging two companies with incredibly different cultures and overlapping products, so the value to be gained from the deal is at a significant discount.

The question has to be asked: what will the Yahoo deal do for Microsoft as a company? The answer: not a helluva lot. I'd imagine most of the really good Yahoo developers will not be too keen on working for Microsoft, so we can expect a raft of defections. Then Microsoft will have to shut down and merge various divisions, paying massive retrenchment packages. The final result: an even more bloated and top-heavy Microsoft with a few niche internet products and one or two good ones (flikr for example).

Many years ago I worked at a large South African IT company with management who'd seriously stuffed up. It seemed that, in order to hide from the shareholders the magnitude of their mistakes, they engaged in acquisition after acquisition. You see, due to all the merger activity, it was incredibly difficult to compare one year's performance with another's, since the company profile had changed so much. Needless to say, the constant, poorly thought out acquisitions were part of the problem, and eventually resulted in the company all but collapsing.

What are they doing?

Given Steve Ballmer's Bush-esque charge for this Yahoo deal, ignoring overwhelming opinion that it's a bad idea and seemingly paying attention only to that tiny group of people who think it's a good idea, one has to ask what his motives are?

  • Is he honestly convinced that such a deal will suddenly turn Microsoft into a Google-killer? If so, the shareholders need to swiftly find a CEO who actually has his feet planted in reality rather than cloud cuckoo land.
  • Does he believe that advertising revenues are going to grow so huge as to turn Google into a Microsoft-killer, able to wipe out all of Microsoft's products at a whim? In this case, he needs some meds for his paranoia. He could also look at trimming down the massive bureaucracy that Microsoft has become, weeding out the time-servers and turning Microsoft lean and mean again. Steve needs to ask himself how successful Big Blue was in wiping out Microsoft back in the day.
  • Is he hoping that this deal will obscure the upcoming financial disasters due to mismanagement of the Windows stack? Whilst this is my latest pet theory, it doesn't really hold water, unfortunately.

Basically, I'm concerned that Microsoft is making misstep after misstep, from Vista itself and the Vista-capable debacle, to the Zune, the hugely overpriced Facebook deal to Yahoo, and many, many others. As a .NET developer I have a vested interest in seeing Microsoft succeed. Sometimes that means chiding them for their mistakes. However, I am growing increasingly concerned about the direction Microsoft executives have been taking recently. If I was Google, and had to pay someone to destroy Microsoft, I'm not sure if I could get them to do a better job than the current rash of poor decisions.

My advice to MS

I wish Microsoft would stop panicking about every upstart industry or internet fad that comes along, and focus on their products. Get Windows to where it should be, and you can probably kiss Apple OS and Linux goodbye. Work with hardware manufacturers well enough and you could possibly kill off MacBook too, or at least relegate it to a niche market. Drop the Zune, let Apple keep that market, with enough care you can effectively turn them into a consumer electronics company like a Sony. In other words, make their flair for design other people's problem, not yours. Make a decent online-capable rich client Office, along with web versions, obviously not as full featured, and you can kill off any possible Google Documents challenges.

Forget advertising, it's a niche and fickle market. Let Google have it, there's already some rumblings of discontent amoungst advertisers around Google's opaque pricing strategies. Let them deal with that pain. Sure they're going to control search for a while to come, which means they will effectively be everyone's landing page. So what?

Oh, and the #1 strategy? Install an ad blocker in IE, and offer to switch it on when the users install IE. You know, something like "Adverts can slow down your browsing experience, and distract from the content on the page". Something like that. Make sure it blocks Google ads. Yeah, sure you're going to effectively de-monetise the web, but since your web efforts have been, ahem, well, not as successful as you'd like, who the hell cares? So half the internet companies subsisting only on advert revenue would shut down almost immediately, why the hell would this be your problem? Best of all, make the ad blocker easily replaceable and you probably wouldn't even have to worry about anti trust issues, especially if you gave up on competing on search.

A friend of mine is looking fo ASP.NET developers. He's basically looking for mid-level to more senior people, who know their way around the frameworks. The person would have to be a self-starter able to work without constant supervision.

If you're interested, his name is Peter Wright and you can get hold of him on (082) 610-0605.  

Dunno how many of you read John Lilly's rant about Apple pushing out their Safari browser via iTunes update the other day? Basically he was rightly upset that Apple were using what should be an update and patch delivery mechanism to subtly trick users into installing their Safari web browser. His main point was

Apple has made it incredibly easy — the default, even — for users to install ride along software that they didn’t ask for, and maybe didn’t want. This is wrong, and borders on malware distribution practices. 

I couldn't agree with him more. The issue is not so much that they deliver Safari via Apple Update, but that it's checked by default every time. The amusing thing is the usual strident defenses from Mac fanboys:

...You Windows users are not only paranoid, but so anti-Apple that your comments are hysterical...

Much to do about nothing! The problem here is clear: Mozilla’s CEO is intimidated by the competition.

Mountain out of a molehill… at least Safari is standards compliant.

I think the problem here is that Safari out performs both IE and FireFox. This is just sour grapes. I would like to hear more about the performance differences.

 What makes these vociferous justifications all the more poignant is the recent failure (again) of Apple in the PWN 2 OWN hacking contest. Now, for two years running, Apple has finished dead last due to security glitches in Safari, the very same Safari they are so eagerly trying to force down the throats of their iTunes customers.

Let me give you a brief synopsis of the hacking contest:

  • Day 1: Three machines (Mac, Windows, and Linux) are placed on the network, and left for the entire day with no user interaction for contestants to attempt to crack. Prize for succeeding: the laptop you crack and $20,000.
  • Day 2: The organisers use various default installed client-side applications. Basically the attackers can ask the organisers to open emails, or visit web sites, but not to open attachments or install programs. Prize: laptop plus $10,000.
  • Day 3: Much more open, some popular third party apps will be installed and used. Prize: laptop plus $5,000.

The Mac fell 2 minutes into Day 2. As of this time neither the Windows or Linux machine have been compromised, with Day 3 still to go [Update - Vista fell 7 hours into Day 3]. The Slashdot thread on this is also illuminating, with the most coherent fanboy conspiracy theory the severely addled idea that the researchers wanted a Macbook more than the other laptops. Therefore, they decided not to attack the other machines and concentrated on the Mac, clearly ignoring the fact that $20,000 or $10,000 can buy you quite a few Macbooks. Another idea from the sheep was that the Mac is so secure that the hackers wanted the challenge of attacking it first. The mind seriously boggles, but I guess once you enter the Reality Distortion Field your mind goes blank and you start drooling like a slack-jawed idiot. One of the best responses I saw to one of the fanboys was this one:

Really? Because I see the Mac having come out as the clear loser in a head to head contest on a level playing field against the two biggest competitors it has in the laptop market. Seems pretty simple to me.

Now let me stop giggling and get back to my point, I'm sure I had one. Do I think that Mac is crap because of a Safari flaw? No, not at all. Besides my digression into the vagaries of fanboyism, all it says is that Safari is not very secure compared with IE7 and Firefox. In fact, it's so insecure that twice it's been the cause of Mac failure at this competition.

This is the application that Apple has decided to share with us in an underhanded and sneaky way. They are surreptitiously installing an application which provides known attack vectors into the operating system. More particularly they're supplying this to people who probably have more secure browsers already, such as Firefox or IE. Does this strike anybody as strangely familiar? (Hint: Sony rootkit)

Now sure, this is not at all to the same degree, and there's no guarantee that Safari on Windows is going to be as insecure as Safari on OSX, I just don't think Apple should be slipping this kind of stuff in on the sly is all.

Update

The Vista machine went down 7 hours into the third day due to an Adobe Flash vulnerability, leaving Ubuntu Linux running on a Sony Vaio as the last machine standing.

Now what this means is that the Mac was undone by an application that Apple pre-install on their computers, whilst Vista was undone by a third party application. Given Vista and IE7's vaunted security, it should not be possible to hijack the computer, no matter how many bugs there are in Flash. Nonetheless, Microsoft are not pushing out Flash (or even Silverlight for that matter) silently, and you have to choose to install it, a choice Apple is still deceitfully trying to take away from iTunes users.

Let me say it again, deviously installing software that opens your computer to attack vectors is pretty much the definition of malware. The difference between Apple’s tactics and Sony BMG’s is not a difference in principle, merely one of degree.

What's something everybody wants for their application, but very few people have the time to deliver? Performance. Let's face it, in most software projects, performance requirements are relegated to the very end of the project, when every knows they won't have the time to address them. In one sense this is a good thing, as one of my biggest bugbears is premature optimisation.

Premature optimization is the root of all evil.

- Hoare's Dictum, Sir Tony Hoare

When should I optimise?

Now keep in mind that by denigrating premature optimisation I am not saying that you should never think of performance when writing an