Coding Sanity

Well, I'm sure we've all heard the ruckus about the continuing problems with the eNaTIS system. Due to the issues this system is having, my new car is taking forever to get registered. No complaints here, with my temporary license, I don't have to worry about speed cameras in front of me, giving me plenty of opportunities to really test it's engine capacity.

eNaTIS is being developed by a company called Tasima, a joint venture between arivia.kom, Thuthukani, and Webcom. I'd like to direct your attention to an article from their news section that expresses their glee at the eNaTIS system hitting 12 transactions per second, which they hit by doubling the processing power in their data centre.

 To quote from a paper by the legendary Jim Gray et al, written in 1984:

Ordinary transaction processing systems and techniques bottleneck at 50 transactions per second (tps) while high performance transaction processing systems achieve 200 tps

So, he's discussing systems that outperform eNaTIS by over 400%, but he's discussing those 23 years ago. Let's have a look at Moore's Law shall we? Given a doubling every 24 months we should expect a low-performance system to be able to handle about 51,200 tps, placing eNaTIS in the bottom 0.02% of the performance spectrum. Oracle indicate that to purchase a Windows box capable of performing 25,000 tps will cost about a million US Dollars, so it looks as if a straight application of Moore's law is a bit unfair. A quick Google search throws up a few summaries indicating TPS counts in the tens of thousands for fairly high-end systems at least.

The Transaction Processing Performance Council's TPC-C benchmark indicates that the best system from a price/performance standpoint would be expected to handle 82,774 New-Order transactions per minute, all processed whilst "the system is executing four other transaction types". That works out to a bit under 1,400 transactions per second. If you ignore the price you can get up to just over 68,000 tps. This means that if we take the cheapest option, eNaTIS is still only at 0.86% of the benchmark performance.

It's of course not generally a good idea to compare production systems with benchmarks, but with these kind of figures there is clearly something very wrong with being proud of 12 transactions per second.

Performance is too often relegated to the dustbin, only worried about at the end of a project lifespan. Avoiding premature optimization is not a license to ignore performance issues entirely. A critical part of specifications and tests should be the performance expected from the systems.

Maybe I'm being too harsh on eNaTIS. It's possible that there is some huge processing hurdle that they just cannot get around that is causing such embarrassingly low performance statistics. I haven't investigated their system much. I will say that any CV that appears on my desk from Tasima will be getting a seriously hairy eyeball.

Update 8 May 2007

Well, eNaTIS is still experiencing major troubles.which appear to have to do with server capacity, which are causing service failures at licensing departments around the country. I fail to see how this can be reconciled with Johan Vorsters comment below indicating that the speed is merely a function of how fast the users are working, and is not limited by the system. The various DOT spokesmen appear to have a different take on the matter, blaming the situation on server capacity.

However, there also may be architectural issues, from IOL, Colin Msibi the national transport spokesman said, "The server has been installed. Technicians will work through the night on the communication between the server and the data system". I dunno about you, but generally that's just a high-speed backbone, you plug the server in to your backbone and communication is working. I'm simplifying a touch, but I don't see it taking hours.

As pure speculation, based on my personal experience, it seems to me that this project bears the hallmarks of having been designed in a client-server architecture rather than a properly distributed one. Highly chatty interfaces lead to exactly the kind of instability we are seeing, and processing "live" instead of using message queues can cause huge system slowdowns as the server is forced to process work now that could easily have been deferred to later.

Update 19 May 2007

Another week, another set of staggering embarrassments for Tasima and the DOT. Earlier this week there was the 702 exposè behind the scenes at a licensing station. I did not hear it, but by all accounts they were able to confirm that the system is horrifically slow, taking up to an hour to process transactions that used to take minutes. Numerous lock-ups and crashes were reported. Tasima appears to be blaming the problems on old Telkom lines. This just reinforces the importance of performance testing, which was clearly not done to any significant degree. Such testing would have clearly identified:

1. Required server resources
2. Required client resources
3. Bandwidth requirements

 Since all of these are being changed "on the fly" to try an cope with the load, it is abundantly clear that if any performance testing was done that it was highly inadequate.

The eNaTIS web site is now announcing a breakthrough of 18 transactions per second, and that only 0.2% of transactions take more than 1 minute to process. This is clearly not what was experienced by 702, and it is in fact worrying that any transactions take a minute to process. In addition, in their discussion of the transactions, it now becomes clear (since they claim that "n most cases, transactions are processed within 9 seconds, even during peak hours") that the transactions they were talking about are system transactions, not user-driven ones, giving the lie to Johan Vorsters comment on this topic below.

Finally, in yet another slap in the face to Tasima, the DG's audit report on eNaTIS was allegedly acquired by the Beeld, and has some interesting points in their article on News24. Of 24 apsects of the system, the DG found 19 of them to be high-risk. More worrying, they also highlight that this new system has significant security issues.

 I've changed my mind, CV's from Tasima will go straight into the trash.

Update 1 June 2007

The Beeld has a report about the security issues with eNaTIS system. Very worrying was that one did "not need a password to log on as an ... administrator", that the "documents ... are not secured", and "can be circulated [without any protection]". A Professor Basie von Solms said "It's not just irresponsible, but possibly also criminal to keep operating a system with so many shortcomings,". He also seems to believe that these issues would be apparent to first year students. This a begs the question of who exactly the developers were who implemented eNaTIS? Did they have any formal development accreditation? Did they have much experience? Were they just hired off the street, no questions asked, and billed out at outrageous rates? I ask the last because I am aware of at least one major South African development company that appears to have that as their modus operandi. I'm now wondering whether arrvia.kom/Tasima have the same strategy?

Just when you think it can't get any worse, how about this quote from the article "Werner Koekemoer, project manager of eNatis, laughed outside court when he was asked if the security shortcomings pointed out by the A-G, had been rectified."

Why is he laughing? Because the issues have not been rectified? If so, it's hardly a laughing matter one would think? Is he laughing because they have been rectified? But laughing would be a strange response in that case, one would expect something along the lines of, I don't know, maybe, "YES!". So, it looks like our entire traffic information system is wide open to attack, and the project manager appears to find this amusing.

Now where did I put that copy of Hacking? Time to excise any records of speeding fines I think.

Just some more fun facts:

• eNaTIS is using free DNS.
• On the 25th of May the admin log was in full public view at http://www.enatis.com/administrator/, according to a mybroadband user called Telkomisaloser. Unfortunately, it is now access controlled. Anyone want to have a go guessing the admin password? I'm guessing a dictionary approach would work quite well.
• Face Technologies (the arivia.kom component), is a member of the Probidia consortium which also has Schabir Shaik as a member. Great, they're in bed with a convicted felon. In fact Johan Vorster , whose comment can be found below, the MD of Tasima is still an executive manager of Probidia.
• Cool quote from sabagamma (also on the mybroadband forums): "Arivia.kom's involvement is frightening - everyone wants to buy them out but nobody wants their staff. That says something considering that IT is mainly built from human capital."
• They appear to now be running on 4 Oracle database servers. That means that each server is handling about 3 transactions per second!

According to SuperCricket, "Graeme Smith believes his team have finally buried their chokers' tag after storming into the World Cup semifinals".

No, they will shed that tag when they win the World Cup final. And since when is losing to Bangladesh and then hammering England called “storming”? I’d call it inconsistent. Especially considering that we're the bottom placed of the semi-finalists.

We're going to need significantly more reliability than this to beat Australia.

There are several online newspapers carrying a story purporting to be from Outsurance, claiming that robbers use rubbish outside your property as a mechanism for marking which places to rob, and which to leave alone. You can find the story here, here, here and here. Don't bother following each link though, since the stories are pretty much word for word. The story is sourced from SAPA-DPA, which means it comes from Deutsche Presse-Agentur, a German news organisation. Well, I found a couple of things suspicious about the story.

Let's think for a moment about this alleged behaviour. First, does anyone honestly think that a thief would use something as ephemeral as rubbish lying outside someones property as a means of keeping track of cased houses? What if the gardener cleaned it up? What if someone switched it? What if the wind blew it, or a dog grabbed it? I'd be willing to guess that most robbers, needing to keep information about potential targets would use, um, note paper? Not difficult to use, we've only had writing for a few thousand years. Even an illiterate would be able to make meaningful marks, and I submit that any set of robbers sophisticated to case the places well beforehand would have at least one literate member.

Secondly, let's consider that using a publicly visible sign means that other criminals now effectively have access to the "intelligence" that the first thieves worked hard to obtain. So, one could start a very successful gang in theory that could drive around looking for the "easy" signs, and rob those houses, and never have to do any research or investigation oneself. Please! Criminals working together? Colluding? These are asocial people at best. I'm not saying that criminals don't sometimes work together, but this implies that every single housebreaker in South Africa is part of a massive, coordinated organisation.

Thirdly, have a look at the burglary figure, it claims to be for the March 2005-March 2006, but the SAPS have only released statistics for up to 2003/2004 (that I could find anyway), and the burglary figures were much higher than that, hell the only province with figures lower than that was Limpopo. This is a completely made-up figure.

Fourthly, consider that there is no person from Outsurance attributed. Normally such large corporations have spokespeople who insist on having their names in the story when they're quoted. This is partly to make it easy to chase them up. I called Outsurance this morning and actually spoke to their spokesman who confirmed that this story does not come from Outsurance at all.

So, what we have is a bogus story that assumes that South African criminals are stagerringly stupid, illiterate, and conspiring together. Hmm, what stereotypes does that fulfill? You know, the problem with President Mbeki's rants about racism being alive and well, especially in the debate about crime, is that sometimes he's right. Whoever wrote this story was a complete racist, and all of us who accepted it without thinking (and I certainly did at first) at the very least still harbor some powerful negative stereotypes. Nothing wrong with that, as long as we recognize it and just practice engaging our brains long enough to challenge them.

Stories that pander to our preconceived notions or fears are like viruses slipping past our defences. The only way to ensure one is not hoaxed again and again is to question everything we hear, no matter what the source.

Update 8 May 2007

Despite numerous requests to DPA, to Diago, to Kat and some others offline, I have still not received a single original source. It's always someone else. Everyone promises me sources, but never deliver. So, at this point, I have no qualms whatsoever as marking this story as a complete fabrication and urban legend. Everything so far has been hearsay and innuendo. It's actually an object lesson in why courts do not admit hearsay testimony. Despite the strong feelings running about this story, not one person testifying to it's veracity can actually back up their claim. Not one.