Wednesday, November 09, 2005

Planner Mode for EMACS

I came across Planner mode a couple of weeks ago. It has been working very well for me. It integrates with a variety of other EMACS applications. It is also able to keep track of time spent on a given task, which is handy for a number of uses.
In the past I have used a variety of contact management/calendaring software including Outlook and Evolution but they are big and bulky and get in the way. They are also overly complicated and it is hard to associate notes with a given piece of information.

Wednesday, September 21, 2005

Useful web site

This site has a lot of interesting articles on a number of different languages.

Thursday, August 18, 2005

Very Interesting Lisp Primer

It gives a very nice history and current state of Common Lisp as well as pointers to a lot of additional material. Pascal Costanza's Highly Opinionated Guide to Lisp

Monday, August 08, 2005

Find 'hidden' resources in extension DLL's

While creating your new, nifty grid control for your MFC extension DLL, you put your resources into your MFC extension control. Then when you compile and run your test application, or worse, your production application, it can’t find the controls resources. A worse problem is that it silently uses the wrong resources.

Identifying the problem

The problem is that MFC assumes that all of the applications' resources reside in the main executable. There are two ways to solve this problem. The first method is to let MFC hunt for the correct resource using AfxFindResourceHandle. The second method revolves around treating the extension DLL like a separate resource DLL.

However, there are significant disadvantages to both methods. The main disadvantage with the first method, which uses AfxFindResourceHandle, is that there can’t be any duplicate resource IDs in any of your applications DLL’s. Also, this method is slower.Both occur because AfxFindResourceHandle searches the entire link chain in the application and returns the first module with a matching resource.

The primary disadvantage with the second method is that you have to explicitly get a handle to the target module; and since loading each kind of resource uses different functions, you end up with code that is almost identical for each kind of resource.

Ask MFC politely for the resource

To have MFC do all the heavy lifting, call AfxFindResourceHandle, like this:


This will work fine--until someone adds an icon with a duplicate ID to the code base. There is one advantage to this method of loading a resource: You don’t have to know which module the resource resides in.

When something can go wrong, it will.

Being a firm believer in Murphy’s Law; I like to be as explicit as possible. So I prefer the second method of loading resources in my extension DLL’s. Although it requires a little more investment upfront, it pays off when your code goes into maintenance mode. Basically, for each kind of resource you use in your extension DLL, you’ll write a function like this:

CString LoadResString(UINT ResID, CString strModuleName)
TCHAR szBuffer[256];
//The module handle can be obtained in a variety of ways the
//following is the most
//straight forward
HMODULE mod = ::GetModuleHandle(strModuleName);
if (!mod){
AfxMessageBox(_T("GetModuleHandle failed in LoadResString."));
RaiseException(1814, 0, 0, NULL);
//This is where the resource actually gets loaded and will be
//different in each function
int count = LoadString(mod, ResID, szBuffer, 255);
if (!count){
AfxMessageBox(_T("LoadString failed in LoadResString."));
RaiseException(1814, 0, 0, NULL);
CString rslt(szBuffer);
return rslt;

MFC extension DLL’s are a nice feature but because of all the legacy code that Microsoft has to support we are stuck with some “interesting” features like this problem with MFC not looking in the extension DLL for resources first. It’s an easy problem to overcome but it can give you a headache the first time you encounter it. It doesn’t help that you have to know what the problem is before you can find the answer in the documentation.

Note: Technically, there is a third method for specifying which module to load resources from. I chose not to examine it in this article because it's an even less explicit than the AfxFindResourceHandle method.

Friday, August 05, 2005

Marking an ActiveX object safe for scripting

I recently ran a cross a problem where I had to create a COM component for use in Internet Exploder. In order to mark component safe for scripting you need to add the following to your Python class.

_reg_catids_ = ["{7DD95801-9882-11CF-9FA9-00AA006C42C4}"]

Now the class won't cause Internet Exporer to prompt for permission to load the component. I wonder why there is such a spyware problem? :0

Thursday, August 04, 2005

Event logging the .NET way

Those of you who have used the event logging API are in for a pleasant surprise. In the past, event logging required (comparatively) arduous coding. After creating an executable with specially formatted resources to be used as a message source, you had to register the message source. Then you finally got to write to the event log.

The System.Diagnostics name space contains (among other things) the event logging API. The two classes we'll focus on for reading and writing to the event log in .NET are the EventLog and the EventLogEntry classes.

Write to the event log

Writing to the event log in .NET follows the same basic procedure as using the Win32 API. First, open the event log.

//The period means this machine
EventLog log = new EventLog(“Application”, “.”, “MySource”);

Then write the event:

log.WriteEntry(“Test message”, EventLogEntryType.Information);

Read from the event log

Reading the event log the Win32 way requires you to open the log with the OpenEventLog function and retrieve a handle to the Event Log. Then you have to retrieve the EVENTLOGRECORD structure with the ReadEventLog function.

The EVENTLOGRECORD structure has both static and dynamic fields. In order to access the dynamic fields of the structure such as the message and the source, you have to treat the structure like a byte array. To make things inconsistent, you access the static fields with the dot operator like normal. Although it’s not difficult, it can be error-prone, and if you miscount bytes, you'll get access violations or bad data.

Reading the event log the .NET way is much improved. Instead of dealing with an EVENTLOGRECORD structure, you get to deal with a much safer and easier EventLogEntry class. When you create an EventLog instance pointing at a specific log, the Entries property (which is an instance of the EventLogEntryCollection class) automatically fills with EventLogEntries. You can use the following code to access the Entries collection:

foreach EventLogEntry entry in eventLog.Entries{

Create a custom event log

You can also create a custom event log by calling the static method CreateEventSource of the EventLog class. For example:

//Check to make sure that the source that will
//be associated with the new log doesn’t exist.
//SourceExists is also a static method
if (!EventLog.SourceExists(“MySource”){
    //If MyNewLog does not exist it will be created.
    EventLog.CreateEventSource(“MySource”, “MyNewLog”);

As the above code indicates, you can check to see if a source is registered. If the source name already exists, you could overwrite someone else’s source, creating havoc.

Compare the previous code to what you would have to do to create a custom event log with the Win32 API:

  1. Create a message file.

  2. Compile it using the message compiler.(Be sure to include the created resource into the resources of the message synch DLL or Executable.)

  3. Register the message synch (which is about 100 lines of code) in the event system by creating the requisite registry entries.

  4. Take a look at the keys under HKLM\System\CurrentControlSet\Services\EventLog in the registry.

  5. Open and retrieve a handle to an event log.

  6. Now you can write the custom event log.


As you can see, event logging the .NET way is quite easy. The only drawback is that your application has to be running on a version of Windows based on Windows NT. But since Windows XP is the new consumer OS, this issue will soon be moot.

Monday, August 01, 2005


When dealing with the Windows event log, a developer usually only wants to write events to the log. The Event Viewer, on the other hand, is good at displaying event log messages and lets you view, sort, or filter these messages from a local or remote machine. However, sometimes it's necessary to read events from the log. For instance, if you're dealing with end users and you want the users to be able to click on an icon and have all relevant event log entries sent to you via e-mail.


If you look at the documentation for the event-logging API, it appears that reading entries from the log is as simple as this.

DWORD nBytesRead, nBytesMin;
HANDLE hEntry = OpenEventLog( NULL, "Application");

0, &record, sizeof(EVENTLOGRECORD),
&nBytesRead, &nBytesMin)){
//Do something with record


There are a number of problems with the preceding code, which stem from the fact that EVENTLOGRECORD is a dynamic structure. If you look at the documentation for EVENTLOGRECORD, you'll see the following:

typedef struct _EVENTLOGRECORD {
DWORD Length;
DWORD Reserved;
DWORD RecordNumber;
DWORD TimeGenerated;
DWORD TimeWritten;
WORD EventType;
WORD NumStrings;
WORD EventCategory;
WORD ReservedFlags;
DWORD ClosingRecordNumber;
DWORD StringOffset;
DWORD UserSidLength;
DWORD UserSidOffset;
DWORD DataLength;
DWORD DataOffset;
// Then follow:
// TCHAR SourceName[]
// TCHAR Computername[]
// SID UserSid
// TCHAR Strings[]
// BYTE Data[]
// CHAR Pad[]
// DWORD Length;

All of the fields in comments are just examples. Yet if you try to do something like this:


you'll get a compile error. All of the fields after DataOffset are one big buffer, and you have to treat it as such to work with it. For instance, to get the length of the SourceName field, you must do the following.

//precord is declared as a pointer to a byte buffer.
DWORD length = _tcslen((TCHAR*)precord + 56);

The magic number 56 happens to be the total number of bytes preceding the SourceName field. So if you want to get the value of the ComputerName field, you use the following code:

DWORD offset = 56 + _tcslen((TCHAR*)precord + 56) + 1;
//The +1 is for the NULL byte
DWORD length = _tcslen((TCHAR*)precord + offset)

The other major problem is that there isn't a way to tell how big the EVENTLOGRECORD is ahead of time without attempting to read it. This means you have to call ReadEventLog log twice: the first time with parameters that you know will fail and the second time with the correct parameters. If you look at the last parameter, nBytesMin, this variable will contain the number of bytes necessary to read the requested number of records. If you didn't catch that, you can read multiple records with each call to ReadEventLog. Here's some code that works:

DWORD nBytesRead = 0;
DWORD nBytesMin = 0;
DWORD nBytesToRead = 0;
BYTE *precord = NULL;
BYTE fake[1];

HANDLE hEntry = OpenEventLog( NULL, "Application");

//This one fails but gives us the event record size
0, fake, 1,&nBytesRead, &nBytesMin);

nBytesToRead = nBytesMin;
precord = new BYTE[nBytesToRead];
precord, nBytesToRead, &nBytesRead,
&nBytesMin);//read one record

Now we extract some fields.

CString SourceName(precord + 56);
CString ComputerName(precord + 56 + SourceName.GetLength() + 1);


Even though the event log API isn't the most difficult to master, it isn't well documented or obvious. Yet once you master the EVENTLOGRECORD structure, it's smooth sailing.

Friday, July 29, 2005

Hide data in thestream

Streams are a facility built into NTFS that attaches metadata to files; this facility is similar to the extended attributes in an OS/2 file system. Unfortunately, using the term streams could be confusing because it's so overloaded (especially when it comes to files).

A new definition for streams

A file on an NTFS volume is composed of a primary stream and zero or more secondary streams . The primary stream is the data that you normally access, while the secondary stream(s) reside in parallel with the primary stream. Unlike the primary stream (which is unnamed), a secondary stream has a unique name. The secondary stream can hold any amount of any kind of data. However, streams are only available on the NTFS file system. So if a file with secondary streams is moved to another file system, you'll lose the secondary streams.

Secondary streams are invisible

Secondary streams are invisible to both the Windows Explorer and the console. In fact, the Explorer and the console will (incorrectly) report that the space is free. Streams are only partially implemented in Windows even though it’s been around since Windows NT 3.1. In order for the explorer to be stream aware, you need to install an Explorer add-on.

Accessing secondary streams

You can access secondary streams with the standard CreateFile, ReadFile, and WriteFile Win32 API’s or another API such as MFC’s CFile or executable that uses these functions for low-level file access. To access a secondary stream, append a colon followed by the name of the secondary stream to the file name. Stream names are case insensitive just like file names.

Here's the command’s you use when writing to the stream using the console:

echo foo > bar.txt:title

(Note about the above code: You can’t read the stream back from the console with the Type command.)

Here's the code you use when writing to the secondary stream, title, in C:

DWORD BytesWritten = 0;
const DWORD BufferLength = 3;
HANDLE hFile = CreateFile(“bar.txt:title”, GENERIC_WRITE,
0, NULL, OPEN_ALWAYS, 0, 0);
WriteFile(hFile, “foo”, BufferLength, &BytesWritten, NULL);

Here's the code you use when writing to the secondary stream in Python:

File = PrivoxyWindowOpen(“bar.txt:title”, “w”);

Below are some examples on reading the bar.txt:title stream. Specifically, here's code for reading the stream in C:

HANDLE hFile = CreateFile(“bar.txt:title”, GENERIC_READ, 0, NULL,
const DWORD BufferSize = 25;
DWORD BytesRead = 0;
BYTE buffer[BufferSize];
ReadFile(hFile, buffer, BufferSize, &BytesRead, NULL);

Here's the code you use when reading the stream in Python:

file = PrivoxyWindowOpen(“bar.txt:title”, “r”)

The potential for misuse

If you start writing large amounts of data to secondary streams, you're going to create problems for users of the software. Even though they’ll run out of disk space, their systems will continue to report that there is free space. When you use streams in moderation, they can be invaluable tools for keeping key configuration data out of sight from the user.

But people can misuse streams. For instance, there is a potential for malicious use in worms and viruses. I don't know of any virus scanners that check secondary streams for malicious code. Like I said previously, secondary streams can store any type of data--that includes executable code. Hopefully the anti-virus companies catch this problem before hackers start using streams in their malicious code.

Update your application with BITS

BITS is a new service shipped with Windows XP and later. It allows large files to be uploaded or downloaded in the background. BITS will then notify you when the transfer is complete.

What’s it good for?

Windows XP’s version of Windows update makes heavy use of BITS to download updates in the background while the user is doing other things or while the system is unattended. I would say that following Microsoft’s lead in this is the way to go. You can easily use it to automatically download updates for your apps.

In the past developers have written their own update code or just depended on the user to check the applications web site. Using BITS to update your application gives the user more control and is also more convenient. It also reassures the more paranoid user, like myself, who get nervous when they notice the application they are using keeps going out to the Internet for no apparent reason.

Making use of BITS.

The first step in using BITS is to register a job for the queue.

//make sure to include Bits.h

//error checking has been left out

HRESULT hr = 0;

IBackgroundCopyManager* TransManager = NULL;

IBackgroundCopyJob *pCopyJob = NULL;


//Your threading model here


//Set the impersonation level to RPC_C_IMP_LEVEL_IMPERSONATE

hr = CoInitializeSecurity(NULL, -1, NULL, NULL,




//Create an IBackgroundCopyManager instance.

hr = CoCreateInstance(__uuidof(BackgroundCopyManager), NULL,



(void**) &TransManager);

hr = TransManager->CreateJob(L”DemoJob”,


&ID, &pCopyJob);

//Add files to the job

hr = pCopyJob->AddFile(L"http://MyServer/Path/MyFile.Ext", L"c:\\Path\\MyFile.Ext");

One limitation with BITS that I think is pretty severe is that the content on the server cannot be dynamic. So for instance you can’t specify a URL like this, and have the server give you the right version or have the server notify you that you’re up to date. One simple solution to this is to have your installer setup the new job and delete the old job. So whenever an update is installed the installer removes the old job and creates a new job pointing at the future file name.

Checking the status of a job.

There are a number of ways to check the status of a job. You could poll for the jobs status and perform some action based on the status of the job or you can register to receive a notification of when the job has succeeded or failed. You can actually go one step further and register a program to be executed when the job succeeds.

The last option probably makes the most sense for updating an application. In our example application we have the installer registering a job with BITS and also registers a program with a command line argument consisting of the job ID, to be run when the job succeeds. When our transfer job succeeds our little program runs and retrieves the job information, including the path of the installer and the program runs the installer. Walla, our application is updated.

//Continuing from previous code.

WCHAR wJobID[48];

WCHAR wParameters[257];

IBackgroundCopyJob2* pCopyJob2 = NULL;

WCHAR *wProgram = L”C:\program files\MyApp\UpdateUtil.exe”;

StringFromGUID2(ID, wJobID, 48);

wsprintf(wParameters, L"%s %s", wProgram, wJobID);

pCopyJob->QueryInterface(__uuidof(IBackgroundCopyJob2), (void**)&pCopyJob2);

hr = pCopyJob2->SetNotifyCmdLine(pProgram, wParameters);

//remember to release you COM objects




Wrap up.

One of the first useful axioms I learned when I got into this industry was, “never code what you can steal”. This isn’t exactly stealing but it’s in the same vein. The less code you write, the less code you have to test, and the less code you have to maintain.

Wednesday, July 27, 2005

Taking Advantage of COM with Python

Microsoft's Common Object Model (COM) implements a cross-language object model.COM is the basis for most of Microsoft's recent technologies, such as WMI and Microsoft Office. COM allows you to write code in one language, like C++, and use it in another, such as Python or VB. Python is capable of both using COM objects and creating COM objects. It's been around, in one form or another, for a number of years.

Python has excellent support for COM. In fact, Python's support for COM is so good that COM can be used as a legitimate way to extend Python. If you need to write a Python extension that will be running exclusively on Windows, strongly consider using COM. COM objects can be reused, providing flexibility and reusability, as well.

To use COM in Python, you must install the Python Win32 extensions, which can be downloaded from The pywin32 package comes in a nice installer package, so compiling isn't necessary.

Using a COM component

The steps for using a COM object is the same for all languages that support COM. First you must instantiate the object. The instantiation can be directly incorporated in the language like Jscript's ActiveXObject function or it can be supported via a library, like Pythons Dispatch function. The only difference between the two approaches is that with library based support you have to import the library code first. Once the object is instantiated, you can use the properties, fields and methods just like they were part of normal Python objects.

To start, run the makepy utility on the component that you intend to use. This isn't a mandatory step, but it has some nice advantages. Specifically, named constants defined in the typelib will be available to you when you import the constants class, and if you're using the PyWin IDE, you'll get intellisense for the properties and methods of the component.

The easiest way to run makepy is through the PyWin IDE. Launch PyWin, and under the tools menu, choose makepy. This opens a simple little dialog that contains a list of all the registed COM components on the system.

You can use a COM component without running the makepy utility, but you'll then have to use numeric values in place of the named constants that you normally use for parameters and such.

Using a COM component in Python is nearly as simple as using the component in VB.First you need to import the COM support module:

import win32com.client as w32c

If you ran the makepy utility on the component, then the constants class has all of the named constants defined in the typelib:

from win32com.client import constants

Once the module is imported, you can instantiate the COM component simply:

Obj = w32c.Dispatch(r"")

To instantiate a COM component, you need to know the class name of the component. You can find it in the documentation for the component or by looking in the registry for the object, if you know the GUID. However, If you run the makepy utility on the component then both the name of the component and its GUID will be listed in the generated python file.

Once you've instantiated a component, you have access to all the properties and methods of that component. For example the following code calls the load and transform methods of the MSXML control:

xmldoc = w32c.Dispatch(r"Msxml2.DOMDocument")

rval = xmldoc.load(xml)


The file that accompanies this article shows a complete example that demonstrates how to use the MSXML COM component. To use that example, you need to have the MSXML components installed. (MSXML is installed along with Internet Explorer, so it is usually present on most systems.)

Using COM, you can drive other applications such as Microsoft Office. For instance, Python can be combined with MS Word to create document management systems. Python COM integration is extremely powerful in this regard. On most Windows systems there are numerous COM objects installed. You can browse through them using the makepy utility. Anything that shows up in the makepy utility can be used in Python.

Creating a COM component

Let's look now at what may be the coolest part of Python's COM integration.

Python comes with a large library of modules that do not have equivalents in other languages. By using Python to create COM objects, it is easy to expose some of these modules to other languages.

Creating COM objects in other languages, such as C++, requires a lot of code and knowledge. It also requires that you be familiar with IDL, which is a seperate language. So, to create a COM object in C++, you actually must know two languages.

Contrast that to creating a COM object in Python.

class MD5Obj:

    _public_methods_ = ["Update", "Hash", "Hex"]

    _reg_progid_ = "Python.MD5"

    _reg_clsid_ = "{895E2BD0-0DA7-11D9-9669-0800200C9A66}"

    MD5 = None

    def __init__ (self):

        self.MD5 =


    def Update (self, val):



    def Hash (self):

        return self.MD5.digest()

    def Hex (self):

         return self.MD5.hexdigest()

Notice, that there are no non-Python constructs, no IDL, and no funny preprocessor macros —just plain old Python. The code wraps the MD5 module and exposes the methods that it needs. Something else to notice is that this class does not inherit from another class some magical COM class. What makes this a COM class is the three fields, _public_methods_, _reg_progid_, and _reg_clsid_. _public_methods_ informs informs pyWin which methods you want publicly exposed. Usually you want only the methods that are part of you public interface listed in this field. _reg_progid_ is just a string that gives the object a semi-unique name, that can be used to instantiate the object. The _reg_clsid_ field is a GUID (Globally Unique ID) that is the actual unique name of the object. This is also called the class ID. The class ID can be used to instantiate an object but there are various reasons not to do this. It's better to use the program ID for this. There are a number of utilities to create GUID's available on the Internet as well as part of various development environments.

To have the COM object work in a development environment, you have to register it. The easiest way to do this is just create a little code that register's the object when the script file is run:

if __name__=='__main__':

    import win32com.server.register


That's it. Just run the script and the object is registered. No seperate utility, like regsrv32, to worry about. You can't get much simpler than that.

To use the object from another language, like JScript, is just like using any other COM object.

var md5 = new ActiveXObject("Python.md5");

md5.Update("Th\is is a test");



The full example is available in the script.

Wrapping Up

Even though this was a simple example, everything shown here holds true for complex object hiarchies. You have full access to COM's facilities. You can use attributes, read only attributes, methods, and so on. You've got access to it all as well as the full power of Python. It's all easy as (wait for it...) pie.

Consider using Python to prototype your objects before jumping into C++. This gives you the chance to evaluate your object interfaces without having to go through the build and register step each time you change something.

Who knows, you may find that there is no reason to implement the objects in C++ after all.


Using Pyscript

Windows Common Object Model (COM) isn't the only way to integrate Python with Windows. You can also use PyScript.

PyScript integrates Python with the Windows Scripting Host (WSH), Windows' official scripting engine. WSH is interesting because it's scripting language-agnostic, and because it provides an API for adding new scripting languages to the system.

Using WSH, Python has full access many of the core Windows features, such as such as mapping drives, controlling network printers, and remote scripting. WSH also allows Python to be used in login scripts. And since WSH has poor support for creating graphical user interfaces (GUIs), Python is a nice complement. By using pyscript you get everything that the WSH provides as well as all the capabilities and just plain fun that Python provides.

Python's Win32 library has WSH integration, but it's not enabled by default. To enable pyscript you need to run the file. It's default location is C:\Python23\Lib\site-packages\win32comext\axscript\client\ Running this script simply registers Python with WSH. To create a pyscript file, use the pys extension intead of the py or pyw extensions.

(Because of the many exploits involving vbscript and WSH, the registration script does not enable explorer associations. Considering that many network admins are actively disabling explorer associations with .vbs and .js files, this is a smart default. If you would like to have explorer integration, I have included a reg file that will add the explorer integration for you. If you have an odd setup or use an older version of Python you will need to edit the reg file.)

The example script, example.pys, first creates an MD5 object using Python's MD5 module.

import md5

m =


Next, it creates a link, on the desktop to the wscript.exe application. The MD5 hash is appended to the end of the link description.

WshShell = WScript.CreateObject("WScript.Shell")

Desktop = WshShell.SpecialFolders("Desktop")

link = WshShell.CreateShortcut(Desktop + "\\Wscript.lnk")

link.Description = "Link to the Windows Scripting Host" + " " + m.hexdigest()

link.IconLocation = "wscript.exe,1"

link.TargetPath = "%systemroot%\\system32\\wscript.exe"


Sunday, July 24, 2005

Here are two great books on Common lisp.

Practical Common Lisp

On Lisp

They're both available for download.

An interesting Python function for working with Subversion

This is a function I wrote for working with SVN repositories. It works like os.walk.

import pysvn

Utilities for working with Subversion repositories


def walk (targeturl):
Walk through an subversion repository.

dirStack = []
theEnd = 0
client = pysvn.Client()
client.callback_get_login = prompt
client.callback_notify = notify
client.callback_get_log_message = logmessage

ls =, recurse = False)
files = []
dirs = []
for item in ls:
if item["kind"] == pysvn.node_kind.dir:
if item["kind"] == pysvn.node_kind.file:
yield (targeturl, dirs, files)

while len(dirStack):
files = []
dirs = []
base = dirStack[theEnd]
del dirStack[theEnd]

currentDir = base["name"]
ls =, recurse = False)
for item in ls:
if item["kind"] == pysvn.node_kind.dir:
name = endOfPath(item["name"])
if item["kind"] == pysvn.node_kind.file:
name = endOfPath(item["name"])

yield (currentDir, dirs, files)


Obligatory first post

I am a software engineer located in western Washington. I tend to use a lot of different programming languages in my development projects. Since I enjoy learning new languages, this works out well. I also enjoy experimenting with new gadgets. For instance this post is being written from an Ogo, which is a little email and IM only device.