Mark Torrance.com

Main | June 2005 »

April 28, 2005

On Intelligence

I read Jeff Hawkins' book On Intelligence during December 2004, and haven't stopped thinking about it. It really reawakened in me a longstanding goal I've had to understand more about how intelligence works, how the human mind achieves intelligence, and ultimately to attempt to build intelligent agents.

I did my undergraduate work at Stanford in Symbolic Systems, working with Nils Nilsson, Yoav Shoham and Matt Ginsberg. I studied in the Ph.D. program at the MIT AI Lab with Lynn Stein and Rod Brooks, where I wrote my Master's thesis on a simple autonomous robot system with a basic natural language interface: Natural Communication with Mobile Robots.

I think intelligence needs to be embodied. I think many of the lowest level ideas we take for granted are metaphors which are grounded in our perceived interactions with the physical world, as in Metaphors We Live By. Hawkins' observation that much sensory input to the brain is converted into similar temporal sequences of cortical firings, and his theory that our brain incorporates pattern-recognition capabilities which can work across many different kinds of sensory input and patterns in similar ways, helps to reinforce this idea. It might be possible to build an agent which grounds its concepts in other ways (Cyc?), but I think building an embodied intelligence is more likely to result in a conversant intelligence that works more like the way humans do.

I recently attended a discussion held by SD Forum where Barney Pell, Steve Jurvetson and Konstantin Guericke spoke on a panel about the book On Intelligence. It was a lively and interesting discussion, covering topics from the originality of Hawkins' theories to the value of basic science research to the possible sources of funding for this new generation of mind-simulation research.

Technorati Tags: on_intelligence, ai

Steve Jurvetson took issue with a claim in On Intelligence that modular components of these artificial intelligences would be interchangeable or reusable, allowing researchers to build on each others' work. We agreed that an evolved AI could be copied and reused by others, but Steve was skeptical that a module from one AI would be compatible to be included within another. I however proposed that just as a person who loses one sense learns to compensate by reusing those parts + pathways of their brain for advanced processing of other senses, so it's possible that a module of an AI "Al" which knows how to speak another language could be used + leveraged if it were "grafted in" to a second AI "Ben".

I find the book to be quite credible because it makes, and identifies, several falsifiable claims about the way the brain works. I imagine over the next 5-10 years some neuroscientists will be in a position to support or refute these claims.

Some have objected to Hawkins' approach of publishing these ideas in the popular press without trying to get them accepted first peer-reviewed journals. Personally, I think it's great that he was able to capitalize on his fame to achieve this publication -- that wouldn't be true for just anyone. I also think the ideas are big and interesting enough to deserve the treatment he gave them in a popular book.

My only real issue with the book is the way he puts down the whole field of AI which came before him. There are large subsets of the researchers in that field who share his lofty goals of understanding human intelligence by modelling the human brain. Likewise, as he states, there are big subfields of AI which long ago gave up any pretense of trying to do human modelling; instead they are concerned with achieving usable performance in a variety of areas such as automated vision systems, planning, robot control, etc. I don't view those practitioners as harming or taking anything away from the field of AI; at worst they contributed to the field having a poor reputation due to overpromising and underdelivering.

Patrick Winston, at the time the director of the MIT AI Lab, told me in 1995 that his planning software company wrote algorithms which were used for logistics planning in Operations Desert Shield and Desert Storm. He was told by the head of DARPA that the money saved by the military in that one use-case alone was more than the total amount spent to date, cumulative, by DARPA on all AI research. I take this as a great example of the practical consequences which can fall out of a research agenda which is nominally aimed at grand goals.

But I think Hawkins is right to ask us to refocus our energy on a brain-centered model of intelligence, and I think he's got some very appealing theories about how simple, repeated functional units in the brain, with huge amounts of interconnect and feedback connections, can achieve the amazing feats we observe.

As for me, I feel I'm getting closer to understanding intelligence each time I look for connections or similarities in the way I do things in different fields. How do "mental leaps" work? How do I get to learn a piece of music through repetition, and how is it that the first few notes cue me to sing through the rest of the song? Why is my memory for music and certain other patterns very "linear", while following a "train of thought" or "word associations" feels more undirected and associational?

I was also wondering about a model for neural processing which would leverage existing commodity hardware such as AltiVec vector processing units, or graphics processors. It seems like a densely connected neural model, with strengths on each connection and activity levels at each neuron, can be modelled as a large array math problem. This is probably old hat for the Neural Network guys, but I wonder what could be accomplished by running huge nets on desktop hardware...

I noticed recently that Jeff Hawkins' company Numeta posted a job opening for VP of Engineering. Now that's a job I would love to have.

Posted by mark at 12:08 PM | Comments (1) | TrackBack

April 27, 2005

Performance Measurement

Since 1993, I've built a fair number of interactive web applications. One of the most important areas where my company Vinq differentiates our work is by delivering truly stellar performance and interactivity. This article will cover a few of the techniques we use to evaluate and improve the performance of applications we build.

Later I'll follow this up with another article about some of the specific bottlenecks we've found, how we diagnosed them, and how we set out to improve them.

Technorati Tags: perl, mason, performance tuning

Measure, Measure, Measure

Our principal technique is to build into the development and testing workflow a way to measure the performance of our application. In our case, we're using Time::HiRes, a module for Perl which measures times to millisecond accuracy. We have built a utility function with the signature timer(depth,message) that we can call from within any module (these are Mason components). It stores the current timing information in a variable which is global to the Mason process. Then, at the end of our main template ("autohandler", in Mason parlance), we call another function showtimer() we wrote, which displays the timer output in detail.

This looks like the following:

In this way, we can see the detailed timing for many of the internal functions that are called as part of a web request, so we know where to focus our performance tuning effort. This timer() function also serves as a useful "debug" type of printout -- if we want to check the value of a variable deep in a component, we can print it to this showtimer() area so it doesn't just come out in the middle of the page content.

We set up showtimer() so it is only called if &timer=1 is present in the URL, or if the user is logged in and is a member of the "admin" group. This means we don't have to go to any extra effort to enable performance measurement while we're logged in and working with the site.

Recently, we've added code to showtimer() that highlights the timing for any steps which were slow, so we can focus our attention there.

Logfiles

Logfiles are a tried-and-true performance measurement technique. You have your application write to a logfile somewhere, then you watch the log as you execute the app, and try to interpret what you see. A good way to watch the logfile is "tail -f", which lets you "follow" the logfile in a terminal window and see new lines as they are appended by other programs.

You can sometimes identify slow operations using this method. It's essential to have millisecond-resolution timing, as above. This is also the only method you can use to diagnose problems users report which you didn't directly observe yourself, if you can't reproduce the slow performance when you try again.

One problem with logs is that they can be crowded with extraneous information; info resulting from the activities of other users on the same web server, or just debug messages that you're not interested in. This can be cured in 2 ways:

1. Filter the information at the source, by turning off things you're not interested in. Do this by writing to the log within some kind of DEBUG macro or function, which takes a "log level", then increase the log level for the parts of the code you care about. The advantage of this is that you can leave most of the debug info turned off most of the time, which may improve performance since you're not writing to the logfile so much.
2. Filter the information at read-time. I do this by writing Perl/Mason based filters that show selected content from a logfile, based on matching certain patterns. An advantage is that you can change the filtering on the fly, without editing your source code. You can also make the filter show the elapsed time from the start of an operation, even if the timings stored in the logfile are all absolute.

Load Testing and Performance Testing

You can get a certain amount of information about application performance from load-testing or performance-testing, which involves simulating one or many client interactions with your web application by following a script. Performance Testing involves running a scenario through the application once, to measure the time taken for each step. Load Testing is like performance testing, but you simulate many clients, and see how the whole application performs as the load is increased.

I usually just write a performance test script specific to the application I'm testing, because I want to be able to capture specific timing or content information from the resulting pages, and roll those statistics up across many separate sessions. For example, I might make showtimer() output its results in an XML format, then parse and summarize the output in my script. Or I could add specific timing information about each database operation automatically to my global "timer" variable, then output all of this detail in XML to be aggregated by my performance test script.

The most sophisticated version of this I wrote simulated n user sessions with a delay-between-pageviews of d seconds. It automatically staggered the start times by d/n. It kept statistics on the average, max, and 95% results for the total page timing, as well as tracking those same stats for several of the sub-page timings we care about.

I had a big "aha" moment at StockMaster when I found that for a single-server application (all tiers running on one server), if your showtimer() says it took 250ms (1/4 second) to prepare and draw a page, then your capacity is probably constrained to 4 pageviews per second per CPU. There can be other bottlenecks if you're disk or I/O bound, but by and large this rule of thumb holds true.

What this means is that if you can cut the time for a page from 500ms down to 250ms, you've probably doubled the capacity of what you can get from the same hardware. Not to mention that you've improved the perceived user experience by at least a little bit (it may be dominated by the latency over the internet, or the bandwidth at the end client, which suggests optimizing the size of your content instead).

In my experience, MANY COMMERCIAL WEB APPLICATIONS are poorly optimized. This may be because they are overly complex multi-tier J2EE servlet-based systems running on several distributed servers, or it may be just that their programmers are not that good at performance tuning, or haven't taken the time to do so. In our case, when we're helping our client evaluate commercial solutions and making a build-vs-buy recommendation, we are often agreeing with the client that a commercial solution "feels slow" compared to the apps we build. I don't think it is the J2EE architecture as much as it is details of how the application fetches, processes, caches, and moves data, including the database architecture, proper use of indexes and clustering, testing and measuring each SQL statement, and avoiding redundant work or work that will be thrown away. I'll cover this more in a future article.

Posted by mark at 02:48 PM | Comments (0) | TrackBack

Dotproject

We spend a lot of time managing the projects we implement for our clients. We find ourselves managing tasks, tracking bugs, planning releases, setting priorities, and needing overall reporting. We have as many as 8 developers + 2 QA testers/engineers working on these projects, and 3-4 external customer-employees + our internal project/product manager; coordinating their efforts on a list that has included over 1600 individual tasks over the past year has been a challenge.

We searched for a tool that would let us manage this work with a useful level of detail and process flow. We wanted all of these features:

• Project level reporting with deadlines and priorities
• Tasks and assignments
• Reporting for customers and for product/project managers
• Time logging for billing
• Bug tracking with CVS integration
• Release management
• Monitor external customer feedback

We explored Bugzilla, XPWeb, PHPProject, and Dotproject, all of which are open source solutions which address some of these requirements. In the end, we decided on Dotproject, but we have customized it fairly heavily for our needs, by adding the following:

• CVS integration -- developers are required to provide a Task ID for all checkins they do, and their comments become part of the Task Log for that task, with a timestamp and a link to the CVS diff summary of what they changed. This is also a convenient way for developers to log their hours without needing to access the web-based tool.
• Additional fields of information about tasks including the "release" the task is scheduled to be part of, and a workflow-supporting field called "State" which includes everything from "1 - building a case" to "8 - in QA" and "10 - Released".
• A new category of task called Bug with additional fields Severity and Resolution such as Fixed, Wontfix, CannotDuplicate, etc.
• A new reporting tab that provides aggregate and detailed reports of time spent on various tasks and how those roll up into projects, including pie charts made with jpgraph, sortable columns, and date-range filters

We've found that our modified version of Dotproject, while somewhat complex due to its power, is exactly what we need to replace the mix of Wiki, Bugzilla, and email that we had previously tried to use for the management of this project. Kudos to the Dotproject development team!

Posted by mark at 11:39 AM | Comments (1) | TrackBack

Welcome to MarkTorrance.com

Hi! After a healthy dose of nudging from Barney Pell, I finally decided to start writing a weblog.

For those of you who don't know me, I'm an entrepreneur, company-builder and serious Perl/Web hacker. I'm currently the founder and Principal of VINQ, a consulting company focusing on the creation of large-scale interactive web-based applications such as KnowledgePlex.org and DataPlace.org. Both of these applications were designed and built by Vinq (and our subcontractors Placebase and onlinefocus) for our customer, Fannie Mae Foundation. I was previously the founder of StockMaster.com, the first financial service on the web, which we sold to Red Herring Magazine in March, 2000.

I plan to write about both business and technical issues, and hopefully to help some readers imagine new possibilities in the creation of their own large-scale web projects. I'll start later today with a technical article about the performance tuning I'm currently working on for our new version of DataPlace.

Posted by mark at 11:01 AM | Comments (0) | TrackBack