After a while, successful applications can get set in their ways. Bart Read and Andrew Hunter decided to go for a much more radical approach when given the task of bringing ANTS Profiler up to date, and, almost accidentally, they reinvented the way we do Performance Profiling.
A strange thing happens to applications when they hit mid-life. They are a bit like aging relatives. One is devoted to them but they tend to get set in their ways, complain about having to change, make scathing remarks about the younger generation, and get upset about noise.
ANTS Profiler was one of the first .NET profilers. In its glory days of youth, there was nothing to beat it, but by the time it reached version 3, it was beginning to give the impression of having reached a comfortable middle-age. It began dreaming of comfy sofas and a nice cuppa at the right time in the afternoon, and showing distinct signs of middle-aged spread. In short, ANTS Profiler was becoming more like AUNTie. Something had to be done.
AUNTie and the Un-rest Cure
Andrew and Bart were the ‘enfants terrible’ chosen to shake ANTS Profiler out of its’ complacency by administering an "Un-rest Cure".
An Un-rest Cure, if you don’t know, is named after the famous short story by ‘Saki’
"What you want," said the friend, "is an Unrest-cure."
"An Unrest-cure? I've never heard of such a thing."
"You've heard of Rest-cures for people who've broken down under stress of too much worry and strenuous living; well, you're suffering from overmuch repose and placidity, and you need the opposite kind of treatment."
"But where would one go for such a thing?"
"Well, you might stand as an Orange candidate for Kilkenny, or do a course of district visiting in one of the Apache quarters of Paris, or give lectures in Berlin to prove that most of Wagner's music was written by Gambetta; and there's always the interior of Morocco to travel in. But, to be really effective, the Unrest-cure ought to be tried in the home. How you would do it I haven't the faintest idea”
'
...just a certain glint in his eye'Andrew doesn’t, at first glance, seem like someone who would be able to give an application the Un-Rest Cure’. He is quiet, well-spoken, and never, even in private, does he wear baseball-caps back-to-front. He comes to work, sits quietly, and types code all day long. It was just a certain glint in his eye that gave a clue to the shock that awaited the plump and complacent ANTS application.
"I had an intuitive feeling that ANTS could be a lot faster than it was: almost all of the engine improvements really germinated from that one single idea. Within the ANTS division the changes followed a fairly gradual and logical path but were, ultimately, quite radical. We certainly got a few odd looks and at least one 'you've ruined it!' comment when we showed some of the early results to the rest of the company.”
Bart is noisier and more extrovert than Andrew; he either laughs really loudly or swears really loudly, depending on his current mood. His heartfelt cries of “this is sucking my will to live!” are well-known throughout the office. There was a frisson of anxiety within Red-Gate as Bart and Andrew prepared to wield the scalpel. Were they going to kill the old girl? The original creators of the ANTS Profiler application, Tom and Neil, looked on nervously.
Plastic Surgery and a Boob Job
Much Management restraint was shown, in August 08, when Bart proudly announced that “ANTS Profiler has shed forty pounds and had a boob job…but this is no desperate granny trying to reclaim her lost youth”.
So, is the rejuvenated ANTS 4 more curvaceous than Bart cared to think possible? Do people buy ANTS Profiler 4 or date it? We asked Bart.
...Bart is noisier and more extrovert than Andrew'
“Haha (loud). Well I hope they buy it, but what the customer chooses to do with it in private after that is definitely their own business.
I’m certainly very happy with the appearance, but is it more curvaceous than I cared to think possible? No, not really. Actually not even that close.
We’re still limited by having to use GDI+ at the moment, which really frustrates me. I feel like we’re trapped somewhere in the mid-90s in terms of functionality, and the performance absolutely sucks.
I’d much rather use DirectX because then we could do some much more impressive UI functionality with less overhead but, unfortunately, managed support hasn’t been great.
Managed DirectX never made it out of the beta which expired some time in 2006 and its successor, XNA, is locked in to DirectX 9.0c. Whilst DirectX 9.0c isfour years old, it isn’t something we can guarantee customers will have, and it won’t degrade gracefully under previous versions.
WPF should allow us to do more of the things we’d like to but the .NET 3.5 install base needs to be much larger before we can do that.
With ANTS Profiler 4 we really had to strike a balance. The customer is trying to look at the performance of their code, so it’s really important that the profiler isn’t stealing lots of processor cycles just to paint a few fancy interface elements; an attractive interface that’s enjoyable to use is great, but you have to avoid distracting the user, or detracting from the functionality of the application.
In other words, ‘blinging things up’ for the sake of it is best avoided. However, an example of something that really does enhance the user experience is the call graph animation.
The smooth expansion of nodes makes navigating much easier because the user interface doesn’t snap you to a new location, suddenly leaving you wondering where you are in relation to where you were before.
So it wasn’t just plastic surgery for AUNTie. The team was sketching out some bionics. Andrew takes up the story
"We did try to keep the feel of the old profiler wherever possible, but ended up redesigning the entire interface, which is about as radical as you can get. We added many completely new features: for instance, the call tree that is presented initially uses a lot of data that ANTS 3 simply didn't collect.
Figure 1: The Call tree
The most radical change was probably to the hierarchy view: this became the call graph. The problem with the old hierarchy view was that it would become really hard to trace the path that you had followed, and you couldn't find out how two functions related to each other across more than one level in the stack. The call graph not only allows you to see the path you've taken, but also allows you to explore multiple paths at once, and displays in a comprehensible way how functions relate to each other across multiple stack traces.
Figure 2: The Call Graph
A Very Different Beast
As the bandages came off, Red-Gate’s management looked at the results of the cosmetic surgery and the bionics. They smiled thoughtfully at the result. ‘She’s still the same old AUNTie underneath. Just a bit …err… bigger in some ways and smaller in others, if you understand”.
Little did they realize that, under the covers, ANTS4 was a very different beast from AUNTie. Frankenstein would have gritted his teeth in envy. Like many great innovators, Andrew uses the lame defense that ‘it all just came apart in my hands.’
“We didn't initially set out to rewrite ANTS to the extent that we eventually ended up doing. During the pre-green light process I built some micro profilers that looked at single aspects of the performance of the profiler, and discovered that I could improve performance significantly by replacing some of the core code that dealt with instrumentation. Unfortunately, that meant replacing the code that depended on the old way of doing things, which meant replacing more dependencies, and so forth, until we discovered that there wouldn't be a lot left of ANTS 3 once the changes had been completed.
The performance difference was so obvious with the new system in place that we decided that the effort was worth it. ANTS 3, by default, avoids instrumenting non-user methods, but the prototype ANTS 4 engine wasn't that smart and instrumented everything – and it still beat ANTS 3 many times over in terms of overall program execution time.
Once that decision was taken (with some trepidation), we could look at other improvements we could make along the way. We could look at eliminating time spent waiting for IO operations to complete, improving the accuracy with which we could correct for the influence of the profiler on the program's execution and increasing the amount of detail recorded by the profiler. I think the most interesting change was probably one that was a direct consequence of the way we implemented the speed improvement: we switched to writing out data continuously, so there was no longer any need for the user to manually create snapshots, as the profiler could now produce results for any time range.”
Having suddenly created a dynamic new application, it became obvious that there wasn’t much left of the old AUNT, beloved of many. The team knew that the soul of AUNTie had to be put back. It was painstaking work, as Andrew recalls.
“One of the problems with rewriting an existing product is duplicating the old functionality: I could produce 'version 1' quality features relatively easily, but I had to take account of the expectations that version 3 had built up, and make those old features work with the rest of the new design. This meant that while the initial revision of the new profiler could be written in a fairly short time, there was an extended period of polishing required to make sure it wasn't regressing any important "version 3" features. Overall, it was around 6 months work, nearly twice as long as a typical Red Gate product update.”
Farewell to Snapshots
The final part of AUNTie's unrest-cure was the most radical of all. All those who loved the old application were to get a final shock. All the work of profiling was based on taking ‘snapshots’ of the execution of the application being profiled. The snapshot was part of the application we knew and loved. They went entirely. Why? We asked Bart.
“Because snapshots are a bad solution.
In the past, with slower processors, and slower disks – this is important because of the amount of data we write out whilst profiling – they might have been necessary, although I’m skeptical as to whether that’s really been true for at least the last five years.
Historically, it’s very common for performance profilers to work with snapshots so I think it just falls into the category of something that people have "always done".
Anyway, the problems with snapshots are manifold and are basically these:
- You have to remember to take them. If you don’t, which happens more often than you’d like if you’re doing something complicated with your application, you may find you have to go through the whole rigmarole over again.
- In general a snapshot summarizes all performance data collected up to that point. This is certainly the way it worked in ANTS Profiler 3. The workaround for this is a bit clunky; you have to take a snapshot before you do the thing you want to profile, reset the profiler to clear out all the data, and then take another snapshot after you’ve done the thing of interest. If you don’t do this, you may have a hard time finding the information you want because it’s mixed up with all your start-up code and so forth.
- You might find you want to investigate a particular piece of functionality in more detail, and the only way to do that is re-run the profiler and take more snapshots, which can lead to a lot of going round in circles.
- Snapshots impede the speculative investigation of your code for bottlenecks: you have to have some idea of where the problem lies or how are you going to know when to take your snapshots? If there are problems in other areas of your application that you don’t know about, there’s no way of discovering them, unless you have the time and patience to profile and take snapshots of everything.
During our first design meetings, which included everyone on the team, there was a lot of heated debate around this; I was particularly fed up with the whole snapshots metaphor, to the point of extreme belligerence.”
This was tough talk, but it was all a bit pointless when there was no alternative. While Bart was thumping the table and making references to snapshots sapping his will to live, Andrew was thoughtfully tapping the keyboard…
Hello to the Timeline
Figure 3: ANTS in use, showing the Timeline:
Over the weeks, a smug smile was gradually creeping onto Andrew’s usually-impassive face. It was dawning on him that there was a way of eliminating the need for snapshots.
“It wasn't until we had the initial prototype of the core that we knew that the idea was going to be in any way practical. I felt quite strongly that one of the main problems with earlier profilers was that you had to interact with them to capture the results that you were interested in: you could spend a long time reproducing your performance issue and then forget to take a snapshot – or worse, forget to clear the results recorded during warmup and find that the problem is hidden by a lot of irrelevant code.
Another consequence of this was the idea that we should always try to record everything, where ANTS 3 only tried to capture user code by default. This could result in the profiler failing to record data for something that was relevant, requiring another tedious run through the profiler with new options.”
Bart takes up the story…
“The timeline was so obviously the right approach that there wasn’t any more debate about snapshots. Looking back on those sessions now, what surprises me is how quickly we were able to reach that point; probably on day one, certainly by the end of day two. It really does show what you can achieve with even a little planning.
- You don’t need to remember to do anything with the profiler whilst your app is running. It just collects the data and you drag out the bounds of the time period you’re interested in to see the relevant portion.
- Rather than seeing everything that happened up to a point in time, you can easily filter out everything that happened outside of the time period you’re interested in, which makes zeroing in on the real problem much easier.
- If you want to investigate something that happened in more detail you can just drag out a different region on the timeline. You can also save multiple regions, so if you have several sub-sections of activity related to a piece of functionality, you can investigate them separately.
- If you see something else of interest on the timeline, maybe a prolonged CPU spike that occurred whilst you thought your app was doing something innocuous, you can just drag over it to see what happened. This makes it easy to quickly find and fix performance issues that you had no idea existed.”
Fortunately, nobody quite realized then the effort that it would take. They were soon to realize that a lot of perspiration would follow that moment of inspiration. Bart recalls…
"Not all the challenges were the kind we would have expected. For example, we knew the engine would be a tricky beast, but had no idea that merely collecting data from performance counters would cause so many problems.
We had to completely rewrite the profiler engine to collect and store data in the format we needed, in order to allow people to drag out any region of time and see what happened. Andrew had to do a lot of work over a period of months to get that working.
We also wanted to be able to present performance data in context for the tree view and call graph, and to be able to do so for any arbitrary time period. And it isn’t just that the engine rewrite was a lot of work, but giving the data model the flexibility to allow us to look at the data the way we want is extremely complex, so it’s a tricky beast."
However, all this work paid off because the implementation of the presentation layer of the application turned out to be much simpler, and the initial designs which once seemed so ambitious, proved to be much easier. Bart again…
"From a UI perspective, apart from some issues with synchronisation with the engine, it was relatively straightforward. Painting the timeline isn’t that tricky. The mouse interaction was pretty straightforward, and I nicked the radar control I wrote for SQL Dependency Tracker a couple of years ago to use in place of a standard scrollbar. Apart from being ugly, a scrollbar wouldn’t really help you see what you’d focussed in on relative to other notable features on the timeline. So making sure it looked right and worked right was slightly tricky, but the implementation itself isn’t complicated".
Summary
So, the rejuvenated Aunt returned home from the Health Farm after her Unrest Cure. It is a delicate task to make uncompromising changes to a well-loved application, but now everyone seems to be smiling. Fortunately, the existing user-base was consulted early on, and was able to watch the changes going on. Everyone bought in to the shock-methods of reviving it. It is a radically different application but still recognisably ANTS Profiler – and all the better for an Unrest-Cure.