All posts by cp

Adding a badge to the unified toolbar on Mac OS X

January 21, 2011Cocoa, Development, ITbagde, beta, Cocoa, Mac OS X, NSResponder, NSView, objc, Objective-C, programming, toolbar, unified titlebar, VirtualBoxcp

Mac OS X and the applications running on it are known for being sometimes unusual in the look and feel. For some reason developer on Mac OS X seems to be more creative than on other platforms. For an application developer which deals with user input it is important to make any user interaction as useful as possible and present this information in a way which is on the one side as much less annoying as possible but also attractive and informative on the other side. Lastly I was thinking about how to show the users they are using a beta version, which should remind them that this is not production ready software, but on the same time make it easy to respond to bugs they find in this pre-release. The solution, I came up with, was a badge in the toolbar which shows this is a release which is definitely in a testing phase, but also allows the user to double-click to give a respond to this particular version. In the following post I will show how to do this.

Getting the superview

In Cocoa all user displayed content is a NSView (most of the time; forget about the Dock). This is really nice, cause you could manipulate them. Although there are no public methods for getting the NSView of the toolbar or even the titlebar, they exist as an NSView. The NSView of the toolbar could be accessed by a private method. As always in Objective C you could ask for a particular method by using the respondsToSelector statement like in the following:

NSToolbar *tb = [pWindow toolbar];
if ([tb respondsToSelector:@selector(_toolbarView)])
{
 NSView *tbv = [tb performSelector:@selector(_toolbarView)];
 if (tbv)
 {
  /* do something with tbv */
 }
}

This return the NSView of the toolbar at least until 10.6. But be warned this is a private method and of course this could be changed in a future version of Mac OS X. At least the shown method will correctly fail in the case Apple change his mind which will result in doing nothing. Anyway this will not include the area of the titlebar. To get the NSView which covers both the title bar and the toolbar you need another trick. The titlebar usual has a close, minimize or maximize button. These buttons are NSButton’s and added by the system depending of the window type. As a NSButton is also a NSView it also has a parent. This dependency allows us to access the view which is responsible for displaying the unified titlebar and the toolbar. The following code shows how to get the responsible view and how to add an additional NSImageView to it:

NSView *wv = [[pWindow standardWindowButton:NSWindowCloseButton] superview];
if (wv)
{
 /* We have to calculate the size of the title bar for the center case. */
 NSSize s = [pImage size];
 NSSize s1 = [wv frame].size;
 NSSize s2 = [[pWindow contentView] frame].size;
 /* Correctly position the label. */
 NSImageView *iv = [[NSImageView alloc] initWithFrame:
        NSMakeRect(s1.width - s.width - (fCenter ? 10 : 0),
                   fCenter ? s2.height + (s1.height - s2.height - s.height) / 2 : s1.height - s.height - 1,
  	           s.width, s.height)];
 /* Configure the NSImageView for auto moving. */
 [iv setImage:pImage];
 [iv setAutoresizesSubviews:true];
 [iv setAutoresizingMask:NSViewMinXMargin | NSViewMinYMargin];
 /* Add it to the parent of the close button. */
 [wv addSubview:iv positioned:NSWindowBelow relativeTo:nil];
}

This code needs a NSWindow, a NSImage as label and the flag fCenter for deciding if the image is vertically centered within the titlebar. Also the badge is pinned on the right side by setting the AutoresizingMask. The following shows how this could be look like:

Adding functionality to the badge

First we should add some information about this release by using the setToolTip method of an NSView. Simply add this call to the code:

 [iv setToolTip:@"Some info about the beta version."];

To make this beta hint more useful, we should make it clickable. Getting mouse down events in Cocoa is only possible by handling the mouseDown event of the NSResponder sub-class. Also the view must accept first responder events. How to do this is shown in the following code:

@interface BetaImageView: NSImageView
{}
- (BOOL)acceptsFirstResponder;
- (void)mouseDown:(NSEvent *)pEvent;
@end
@implementation BetaImageView
- (BOOL)acceptsFirstResponder
{
 return YES;
}
- (void)mouseDown:(NSEvent *)pEvent
{
 if ([pEvent clickCount] > 1)
  [[NSWorkspace sharedWorkspace] openURL:
        [NSURL URLWithString:@"http://www.virtualbox.org/"]];
 else
  [super mouseDown:pEvent];
}
@end

Replace NSImageView with BetaViewImage in the allocation call in listing 2 and you are done. Change the URL to one where the user can respond to the beta questions, like a bug tracker or forum.

Conclusion

This post showed how to add an nice looking badge to the unified toolbar on Mac OS X. The user is reminded he is using some pre-release software all the time, but at the same time has an easy way to report problems with this release. Of course could such a badge used for anything else.

Creating file shortcuts on three different operation systems

November 28, 2010Cocoa, Development, IT, QtAESend, Alias, C++, Cocoa, COM, desktop, Desktop Entry Specification, file shortcuts, freeedesktop.org, gcc, hard link, installation, IPersistFile, IShellLink, link, Linux, Mac OS X, objc, Objective-C, platform, programming, Qt, VirtualBox, Windows, X11, Xorgcp

As you may know, developing for multiple platforms is one of my strengths. Strictly speaking, it’s a basic requirement if you are involved in such a product like VirtualBox, which runs on every major (and several minor) platform available today. Beside the GUI, which uses Qt and therewith is portable without any additional cost (which isn’t fully true if you want real native look and feel on every platform, especially on Mac OS X), all the rest of VirtualBox is written in a portable way. This is done by using only C/C++ and Assembler when necessary. Everything which needs a different approach, because of the design of the OS (and the API’s which are available there), is implemented in a platform dependent way. In the history of VirtualBox, several modules are created and grown by the time, which makes it really easy to deal with this differences. For stuff like file handling, paths, strings, semaphores or any other basic functionality, you can just use the modules which are available. On the other side it might be necessary, for a new feature we implement, to write it from the ground. In the following post I will show how to create a file shortcut for the three major operation systems available today.

Why do you want to use file shortcuts

On the classical UNIX systems you have hard and soft links. These are implemented by the filesystem and make it possible to link to another file or folder without any trouble. Most of the time soft links are used, but it really depends on the use case. Unfortunately these kind of links are not available on Windows (yes, I know there are also hard links and junctions on NTFS, but they are not common and difficult to handle), these links doesn’t allow any additional attributes. For example one like to add a different icon to the link or provide more information through a comment field. Beside on Mac OS X, shortcuts can also be work as an application launcher, where the link contain the information what application should be started and how. In contrast to filesystem links which are handled by the operation system, these shortcuts are handled by the window system (or shell) running on the host (which doesn’t mean there is no filesystem support for it). On Windows this is the Explorer, on Mac OS X the Finder and on Linux a freedesktop.org conforming file manager.

Creating a Desktop file on Linux

Desktop files on Linux (or any other UNIX system which conforms to freedesktop.org) is easy. It’s a simple text file which implement the Desktop Entry Specification. In version 1.0 there are 18 possible entries, where not all of them are mandatory. In the following example I use Qt to write these files, but it should be no problem to use any other toolkit or plain C.

bool createShortcut(const QString &strSrcFile,
                    const QString &strDstPath,
                    const QString &strName)
{
 QFile link(strDstPath + QDir::separator() + strName + ".desktop");
 if (link.open(QFile::WriteOnly | QFile::Truncate))
 {
  QTextStream out(&link);
  out.setCodec("UTF-8");
  out << "[Desktop Entry]" << endl
      << "Encoding=UTF-8" << endl
      << "Version=1.0" << endl
      << "Type=Link" << endl
      << "Name=" << strName << endl
      << "URL=" << strSrcFile << endl
      << "Icon=icon-name" << endl;
  return true;
 }
 return false;
}

Replace icon-name by a registered icon on the system and you are done.

Creating a Shell link on Windows

Windows provides an interface for IShellLink since Windows XP. The following example shows how to use it:

bool createShortcut(LPCSTR lpszSrcFile,
                    LPCSTR lpszDstPath,
                    LPCSTR lpszName)
{
 IShellLink *pShl = NULL;
 IPersistFile *pPPF = NULL;
 HRESULT rc = CoCreateInstance(CLSID_ShellLink,
                               NULL,
                               CLSCTX_INPROC_SERVER,
                               IID_IShellLink,
                               (void**)(&pShl));
 if (FAILED(rc))
  return false;
 do
 {
  rc = pShl->SetPath(lpszSrcFile);
  if (FAILED(rc))
   break;
  rc = pShl->QueryInterface(IID_IPersistFile, (void**)&pPPF);
  if (FAILED(rc))
   break;
  WORD wsz[MAX_PATH];
  TCHAR path[MAX_PATH] = { 0 };
  lstrcat(path, lpszDstPath);
  lstrcat(path, "\");
  lstrcat(path, lpszName);
  lstrcat(path, ".lnk");
  MultiByteToWideChar(CP_ACP, 0, buf, -1, wsz, MAX_PATH);
  rc = pPPF->Save(wsz, TRUE);
 } while(0);
 if (pPPF)
  pPPF->Release();
 if (pShl)
  pShl->Release();
 return SUCCEEDED(rc);
}

As you may noticed this uses COM. Many API’s on Windows using the COM interface to communicate between processes. If you don’t use COM in your application you have to initialize it first. This is achieved by adding the following call to the front of the function:

 if (FAILED(CoInitialize(NULL))
  return false;

Depending on your application it might be worth to unitialize COM after usage by appending the following to the function:

 CoUninitialize();

The function itself isn’t any magic. It gets a COM interface to the IShellLink interface and then work with it, by setting the source path and adding a target path by using the IPersistFile interface. As I wrote before you could do much more. Providing a path to a specific application or adding your own parameters is no problem. Have a look at the documentation.

Creating an Alias file on Mac OS X

Shortcut files on Mac OS X are a little bit different. At first, they aren’t one. There are the classical filesystem links and Alias files. Alias files are links which targeting a specific file, but they haven’t all the possibilities of shortcuts like on Windows or Linux. As the name suggest they are really only an alias for another file or directory. So specifying an application to start or things like that aren’t possible. Anyway they allow changing the icon and they are more persistent than on Window or Linux cause they are working with several attributes of the target file. Even if you rename or move the target, an Alias file will resolve the target correctly (if it is possible). On the other side, being such special means also being hard to create. In principle there are two possibilities. The first one is, creating a file which is no file at all, but has several resources forks attached. Therefor you need to know exactly how Alias files are built of and make sure with every release of Mac OS X you are following the development. There is a free project which does exactly that: NDAlias. If you are like me and a little bit more lazy, you ask someone who should know how to create Alias files. This is Finder. Although writing the files itself isn’t easy, asking the Finder to do the job is not really easier, cause the information about doing exactly that are really rare. The following code shows how to achieve it:

bool createShortcut(NSString *pstrSrcFile,
                    NSString *pstrDstPath,
                    NSString *pstrName)
{
 /* First of all we need to figure out which process Id the Finder
  * currently has. */
 NSWorkspace *pWS = [NSWorkspace sharedWorkspace];
 NSArray *pApps = [pWS launchedApplications];
 bool fFFound = false;
 ProcessSerialNumber psn;
 for (NSDictionary *pDict in pApps)
 {
  if ([[pDict valueForKey:@"NSApplicationBundleIdentifier"]
         isEqualToString:@"com.apple.finder"])
  {
   psn.highLongOfPSN = [[pDict
                          valueForKey:@"NSApplicationProcessSerialNumberHigh"] intValue];
   psn.lowLongOfPSN  = [[pDict
                          valueForKey:@"NSApplicationProcessSerialNumberLow"] intValue];
   fFFound = true;
   break;
  }
 }
 if (!fFFound)
  return false;
 /* Now the event fun begins. */
 OSErr err = noErr;
 AliasHandle hSrcAlias = 0;
 AliasHandle hDstAlias = 0;
 do
 {
  /* Create a descriptor which contains the target psn. */
  NSAppleEventDescriptor *finderPSNDesc = [NSAppleEventDescriptor
                                            descriptorWithDescriptorType:typeProcessSerialNumber
                                            bytes:&psn
                                            length:sizeof(psn)];
  if (!finderPSNDesc)
   break;
  /* Create the Apple event descriptor which points to the Finder
   * target already. */
  NSAppleEventDescriptor *finderEventDesc = [NSAppleEventDescriptor
                                              appleEventWithEventClass:kAECoreSuite
                                              eventID:kAECreateElement
                                              argetDescriptor:finderPSNDesc
                                              returnID:kAutoGenerateReturnID
                                              transactionID:kAnyTransactionID];
  if (!finderEventDesc)
   break;
  /* Create and add an event type descriptor: Alias */
  NSAppleEventDescriptor *osTypeDesc = [NSAppleEventDescriptor descriptorWithTypeCode:typeAlias];
  if (!osTypeDesc)
   break;
  [finderEventDesc setParamDescriptor:osTypeDesc forKeyword:keyAEObjectClass];
  /* Now create the source Alias, which will be attached to the event. */
  err = FSNewAliasFromPath(nil, [pstrSrcFile fileSystemRepresentation], 0, &hSrcAlias, 0);
  if (err != noErr)
   break;
  char handleState;
  handleState = HGetState((Handle)hSrcAlias);
  HLock((Handle)hSrcAlias);
  NSAppleEventDescriptor *srcAliasDesc = [NSAppleEventDescriptor
                                           descriptorWithDescriptorType:typeAlias
                                           bytes:*hSrcAlias
                                           length:GetAliasSize(hSrcAlias)];
  if (!srcAliasDesc)
   break;
  [finderEventDesc setParamDescriptor:srcAliasDesc
    forKeyword:keyASPrepositionTo];
  HSetState((Handle)hSrcAlias, handleState);
  /* Next create the target Alias and attach it to the event. */
  err = FSNewAliasFromPath(nil, [pstrDstPath fileSystemRepresentation], 0, &hDstAlias, 0);
  if (err != noErr)
   break;
  handleState = HGetState((Handle)hDstAlias);
  HLock((Handle)hDstAlias);
  NSAppleEventDescriptor *dstAliasDesc = [NSAppleEventDescriptor
                                           descriptorWithDescriptorType:t ypeAlias
                                           bytes:*hDstAlias
                                           length:GetAliasSize(hDstAlias)];
  if (!dstAliasDesc)
   break;
  [finderEventDesc setParamDescriptor:dstAliasDesc
    forKeyword:keyAEInsertHere];
  HSetState((Handle)hDstAlias, handleState);
  /* Finally a property descriptor containing the target
   * Alias name. */
  NSAppleEventDescriptor *finderPropDesc = [NSAppleEventDescriptor recordDescriptor];
  if (!finderPropDesc)
   break;
  [finderPropDesc setDescriptor:[NSAppleEventDescriptor descriptorWithString:pstrName]
    forKeyword:keyAEName];
  [finderEventDesc setParamDescriptor:finderPropDesc forKeyword:keyAEPropData];
  /* Now send the event to the Finder. */
  err = AESend([finderEventDesc aeDesc],
               NULL,
               kAENoReply,
               kAENormalPriority,
               kNoTimeOut,
               0,
               nil);
 } while(0);
 /* Cleanup */
 if (hSrcAlias)
  DisposeHandle((Handle)hSrcAlias);
 if (hDstAlias)
  DisposeHandle((Handle)hDstAlias);
 return err == noErr ? true : false;
}

Although the code above looks a little bit scary, it does not much. It fetch the process serial number of the current Finder process, creates an Application event for creating an Alias file and send this event to the Finder.

Conclusion

Beside showing how to create file shortcuts on different platforms, this article also shows which work is necessary to create platform independent code. It’s a simple example. But it also makes clear that one simple solution for platform one, not necessarily mean it’s such simple on platform two.

Making this easy accessible to any developer is the next step. I will leave this exercise to the reader, but have a look at the platform code of the VirtualBox GUI and the corresponding Makefile.

Understanding some of the mysteries of launchd

November 20, 2010Development, ITApplications, bundle, file types, Info.plist, installation, installer, launchd, LaunchServices, lsregister, Mac OS X, plist, programmingcp

Have you ever wondered how Mac OS X knows which file type belongs to which application? On Windows there is the registry. An installer writes the necessary info into it. Most applications on Mac OS X doesn’t come with an installer, they are just moved from the downloaded DMG file to the /Applications folder. So a developer doesn’t have the ability to take action when the user “install” the application. Anyway there is no need to provide an installer for just this task, cause Mac OS X register file type associations on the first start of the application. In the following post, I will show how to do this, but furthermore I will show where this information is stored and how it could be reseted.

Providing the necessary information to Mac OS X

Applications on Mac OS X need some defined structure. They are so-called bundles, which means on the filesystem layer they are directories. You can prove this by checking the Applications directory within the Terminal.app. You could also right-click on an application and select “Show Package Contents”. The content of the bundle directory is usually hidden from the user when he works with the Finder or any other high level function of Mac OS X (like the open dialog). Additional to this layout on the filesystem, an application provide information about itself to the system with a plist file. This file has to be named Application.app/Content/Info.plist. The following shows exemplary the content of the Tunnelblick application:

<xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>CFBundleDevelopmentRegion</key>     <string>English</string>
 <key>CFBundleExecutable</key>            <string>Tunnelblick</string>
 <key>CFBundleIconFile</key>              <string>tunnelblick.icns</string>
 <key>CFBundleIdentifier</key>            <string>com.openvpn.tunnelblick</string>
 <key>CFBundleInfoDictionaryVersion</key> <string>6.0</string>
 <key>CFBundlePackageType</key>           <string>APPL</string>
 <key>CFBundleShortVersionString</key>    <string>3.1beta20 (build 2132)</string>
 <key>CFBundleSignature</key>             <string>OVPN</string>
 <key>CFBundleVersion</key>               <string>2132</string>
 <key>NSMainNibFile</key>                 <string>MainMenu</string>
 <key>NSPrincipalClass</key>              <string>NSApplication</string>
 <key>NSUIElement</key>                   <string>1</string>
 <key>SUEnableSystemProfiling</key>       <true/>
 <key>SUFeedURL</key>                     <string>http://tunnelblick.net/appcast.rss</string>
 <key>SUPublicDSAKeyFile</key>            <string>dsa_pub.pem</string>
</dict>
</plist>

The keys are mostly self explaining, you can find a full list here.

To register a file type association you have to add an array of the type CFBundleDocumentTypes. Again, here is an extraction of the Tunnelblick application, which shows the registration of the tblk extension:

 <key>CFBundleDocumentTypes</key>
 <array>
  <dict>
   <key>CFBundleTypeExtensions</key>   <array><string>tblk</string></array>
   <key>CFBundleTypeIconFile</key>     <string>tunnelblick_package.icns</string>
   <key>CFBundleTypeName</key>         <string>Tunnelblick VPN Configuration</string>
   <key>CFBundleTypeRole</key>         <string>Editor</string>
   <key>LSTypeIsPackage</key>          <true/>
   <key>NSPersistentStoreTypeKey</key> <string>Binary</string>
  </dict>
 </array>

You have to provide a file extension or mime-type, can add an icon and give a hint what your application can do with this type of file (viewer, editor or nothing).

Beside this passive way of announcing this information, there is also an active way (e.g. for use in an installer). See here for further information.

Where is this information stored

Although Mac OS X hasn’t a registry like windows, some information are stored in global databases, too. Applications are registered at the launchd. The launchd is the central place for starting all kind of programs, from a background service to any common application, like Thunderbird. E.g. background services can register itself and even made their start depending on different events. VirtualBox has an example configuration included, which let launchd start the vbox webservice on activity on a certain port. Although this isn’t used by many users out there (and therefore disabled by default), it shows some generic usage of this functionality.

If you look around in Mac OS X you have the tool launchctl, which allows you to start jobs or register background services with launchd. But it seems there is no tool which is able to get some information about the knowledge of launchd. Well, there is one. It’s a little bit hidden in /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister. To be more comfortable with it, we should include it into the user path by linking it to a known path, like this:

sudo ln -s /Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister /usr/local/bin/lsregister

With this tiny tool we are able to dump the launchd database. lsregister -dump | grep -n10 Tunnelblick displays something like this for Tunnelblick:

10823---------------------------------------------------------------------------------
10824-bundle    id:            6984
10825:  path:          /Applications/Tunnelblick.app
10826-  name:
10827-  identifier:    com.openvpn.tunnelblick (0x800097b8)
10828-  version:       2132
10829-  mod date:      10/31/2010 12:37:08
10830-  reg date:      11/19/2010 11:13:39
10831-  type code:     'APPL'
10832-  creator code:  'OVPN'
10833-  sys version:   0
10834-  flags:         relative-icon-path  ui-element
10835-  item flags:    container  package  application  extension-hidden  native-app  ppc  i386
10836-  icon:          Contents/Resources/tunnelblick.icns
10837:  executable:    Contents/MacOS/Tunnelblick
10838-  inode:         3800129
10839-  exec inode:    3800213
10840-  container id:  32
10841-  library:
10842-  library items:
10843-  --------------------------------------------------------
10844-  claim   id:            17864
10845:          name:          Tunnelblick VPN Configuration
10846-          rank:          Default
10847-          roles:         Editor
10848-          flags:         relative-icon-path  package
10849-          icon:          Contents/Resources/tunnelblick_package.icns
10850-          bindings:      .tblk
10851---------------------------------------------------------------------------------

As you see, all the information provided by the plist file is registered. Beside other information, it also has the registration and modification times stored. To let launchd reread the plist file, usual its only necessary to change the modification time of the application. This could be done by simply executing touch /Applications/Application.app. Anyway, it’s sometimes necessary to reset the content of this database for the own application. This could be done by executing the following:

lsregister -u /Applications/Application.app

To register the application again, just remove the -u parameter. You should also check if your application isn’t registered more than once, with different paths, by using the -dump parameter. To fully reset the launchd database, you could use -kill. This will remove any file type association and registered application. You have been warned.

Conclusion

This article shows how Mac OS X handle informations about installed applications. With this knowledge a developer is able to register the own application into the launchd ecosystem and see how launchd interpret this information. Furthermore, the usage of lsregister allows a developer to analyze the content of the launchd database and make changes to it.

Getting the backtrace from a kernel panic

October 29, 2010Development, ITbacktrace, gcc, graphical modes, kernel, kernel mode settings, KMS, Linux, netconsole, panic, programming, remote access, X11cp

You may know the following situation. You arrive in the morning in the office, do what you always do and check out the latest changes of the software you are working on. After a little bit of compile time and the first coffee you start the just build application. Bumm, kernel panic. After rebooting and locking through the changes you may have an idea what the reason for this could be. A colleague of you is working on a fancy new feature which needed changes to a kernel module. As you almost know nothing about this code you seek for help and, as it of course not happen on his computer, he is asking for a backtrace of this panic. You have two problems now. First you need to see the panic yourself and second it would be nice to get a copy of the backtrace for sharing this info within a bugtracker. In the following post I will show how both aims could be easily archived.

Let the kernel manage the graphical modes

As most people are working under X11 they don’t see the output of an kernel panic. When a kernel panic happens the kernel prints the reason for the panic and a kernel backtrace to the console window and stops immediately its own execution. It is not written into a log file or somewhere else. In consequence you don’t have the ability to look into the panic text, cause the graphical mode is still on. Historically the mode settings are done by the graphic driver of the X11 system. So the kernel has no idea that or which graphic mode is currently in use. Fortunately the kernel hackers invented a new infrastructure which let the kernel do the mode switch. This subsystem is called Kernel-Mode-Settings (KMS). As the kernel do the mode settings, he can switch back to the console on a panic, regardless which graphical mode is currently configured. Beside this, KMS has other improvements like Fast User Switching or a flicker free switch between text and graphic mode. On the other side is this highly hardware dependent and even if it was introduced with version 2.6.28, not all today available hardware can make use of it. If you are an owner of an Intel graphic card you are in good shape. Radeon and NVidia cards have limited support through the in kernel drivers radeonhd and nouveau. For an Intel i915 card you need to enable the following kernel options:

CONFIG_DRM_I915=y
Location:
-> Device Drivers
-> Graphics support
-> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support) (DRM [=y])
-> Intel 830M, 845G, 852GM, 855GM, 865G ( [=y])

CONFIG_DRM_I915_KMS=y
Location:
-> Device Drivers
-> Graphics support
-> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support) (DRM [=y])
-> Intel 830M, 845G, 852GM, 855GM, 865G ( [=y])
-> i915 driver (DRM_I915 [=y])

The kernel line in your favorite boot loader needs the following additional parameter:

i915.modeset=1

X11 should have this minimal configuration for the device section:

Section "Device"
 Identifier    "i915"
 Driver        "intel"
 Option        "DRI"   "true"
EndSection

Please note that you need of course some recent kernel, X11 version and Intel X11 driver to make this work. After a compile, install and boot of the new kernel, KMS should be in use. You will notice it, cause the boot messages will be printed in a much higher graphical resolution, than the usual text mode provide. The next time a kernel panic occurs, the kernel will switch back to the console before the panic is printed. This allows you to see the info printed and maybe you get a useful hint for the reason of the panic.

Post the panic

If you can’t use KMS or don’t want transcribe the panic text by hand into the bugtracker, it would be nice if the text could be made available on another computer. Kernel hackers usual use the serial port for that. Unfortunately most modern computers doesn’t have such a serial port anymore. Also you need two hosts with a serial port and the setup is complex (you have to know about baud-rates, parity and stuff like this). But there is a simpler solution: netconsole. Netconsole is a kernel module, which sends kernel messages anywhere to the net using UDP. The setup is really simple. In the kernel configuration you need the following setting:

CONFIG_NETCONSOLE=m
Location:
-> Device Drivers
-> Network device support (NETDEVICES [=y])

I prefer to compile it as module, which allows me to turn it on only when I need it. Load it with the following command:

modprobe netconsole netconsole=@/,@192.168.220.10/

The ip has to be replaced by the one of your target computer. You can of course tune it much more, like setting source and target ports or even let netconsole send the text to more than one host. On your client you need a network tool which can read from a socket and print the read text to stdout. Netcat or nc are two tools which are able to do just that. The call for nc looks like the following:

nc -l -u 6666

Now if a kernel panic will happen you will see an output like this:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [] rb_erase+0x15c/0x320
PGD 6942f067 PUD a1e4067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/block/md1/dev
CPU 3
Modules linked in: vboxnetadp vboxnetflt vboxdrv netconsole ...

Pid: 18887, comm: VirtualBox Tainted: G        W   2.6.36-gentoo #4 DG33TL/
RIP: 0010:[]  [] rb_erase+0x15c/0x320
RSP: 0018:ffff8800b430db58  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff880069557a68 RCX: 0000000000000001
RDX: ffff880069557a68 RSI: ffff880001d8ed58 RDI: 0000000000000000
RBP: ffff8800b430db68 R08: 0000000000000001 R09: 000000008edcb5d6
R10: 0000000000000000 R11: 0000000000000202 R12: ffff880001d8ed58
R13: 0000000000000000 R14: 000000000000ed00 R15: 0000000000000002
FS:  00007fffde457710(0000) GS:ffff880001d80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000064f9000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process VirtualBox (pid: 18887, threadinfo ffff8800b430c000, task ffff880091e227f0)
Stack:
 ffff88000a03ba18 ffff880001d8ed48 ffff8800b430dba8 ffffffff8105bf06
<0> ffff8800b430dba8 ffffffff8105c97c ffff8800b430dbc8 ffff88000a03ba18
<0> 00004ff8a86ba455 ffff880001d8ed48 ffff8800b430dc48 ffffffff8105ce77
Call Trace:
 [] __remove_hrtimer+0x36/0xb0
 [] ? lock_hrtimer_base+0x2c/0x60
 [] __hrtimer_start_range_ns+0x2b7/0x3c0
 [] ? rtR0SemEventMultiLnxWait+0x250/0x3d0 [vboxdrv]
 [] ? RTLogLoggerExV+0x12f/0x180 [vboxdrv]
 [] hrtimer_start+0x13/0x20
 [] rtTimerLnxStartSubTimer+0x60/0x120 [vboxdrv]
 [] rtTimerLnxStartOnSpecificCpu+0x21/0x30 [vboxdrv]
 [] rtmpLinuxWrapper+0x23/0x30 [vboxdrv]
 [] RTMpOnSpecific+0x99/0xa0 [vboxdrv]
 [] ? rtTimerLnxStartOnSpecificCpu+0x0/0x30 [vboxdrv]
 [] RTTimerStart+0x2a6/0x2e0 [vboxdrv]
 [] ? g_abExecMemory+0x33665/0x180000 [vboxdrv]
 [] g_abExecMemory+0xc678/0x180000 [vboxdrv]
 [] g_abExecMemory+0x328d7/0x180000 [vboxdrv]
 [] supdrvIOCtlFast+0x6a/0x70 [vboxdrv]
 [] VBoxDrvLinuxIOCtl+0x47/0x1e0 [vboxdrv]
 [] ? pick_next_task_fair+0xde/0x150
 [] do_vfs_ioctl+0xa1/0x590
 [] ? sys_futex+0x76/0x170
 [] sys_ioctl+0x4a/0x80
 [] system_call_fastpath+0x16/0x1b
Code: 07 a8 01 75 9d eb 81 0f 1f 84 00 00 00 00 00 48 3b 78 10 0f 84 ...
RIP  [] rb_erase+0x15c/0x320
 RSP
CR2: 0000000000000000
---[ end trace 4eaa2a86a8e2da24 ]---

Normally only kernel panics are sent to the console. You can increase the verbosity level by executing dmesg -n 8 as root.

Conclusion

To continue with the story from the beginning: With the shown methods you can hope your colleague get enough information to find the reason for the kernel panic. To be more helpful, the next step would be to try to debug the problem yourself. Even if the KGDB was merged into the kernel in version 2.6.35, it is not really usable for me. The reason is that it seems kernel hackers usually have really old hardware which either has a serial port, a PS/2 keyboard or both. Otherwise I can’t find a reason why USB keyboards don’t work. I asked on the mailing list of KGDB about the status of USB keyboard support and I can only hope support will be integrated soon.

Using suppression files with Valgrind

October 22, 2010Cocoa, Development, IT, QtC++, Gentoo, glibc, leak check, Linux, Mac OS X, memory checker, memory leaks, programming, Qt, suppression lists, symbol names, Valgrindcp

Valgrind is one of the great tools in the long list of freely available applications for development. Beside several profiling tools it also contains a memory checker. Leaking memory is one of the more common errors a programmer could step into. Basically it means to forget freeing memory (or in a more general sense: any resource) a program has acquired. If you are a perfect developer, this will never happen to you. If you are a good developer it may happen and that’s where Valgrind will save you some trouble. As most of the developers out there are more or less good developers, their programs produce memory leaks, too ;). The right solution for this, is of course to write a bug report. But there are times where this isn’t possible or you are in hurry and don’t want to see all the errors of a third-party library you link against.

In the following post, I will show how to suppress such unwanted error messages to make it much more easier to analyze the output of Valgrind for your own application.

Installing Valgrind

On Mac OS X you can use MacPorts to install Valgrind. You have to use valgrind-devel if you are on Snow Leopard, because Snow Leopard is supported in the current development version only. It’s as simply as typing sudo port install valgrind-devel.

On Gentoo it can become a bit harder. The current stable version is 3.5 (like in MacPorts). If you try this version (at least on an unstable Gentoo like mine) with valgrind ls, you will get the following error:

valgrind:  Fatal error at startup: a function redirection
valgrind:  which is mandatory for this platform-tool combination
valgrind:  cannot be set up.  Details of the redirection are:
valgrind:
valgrind:  A must-be-redirected function
valgrind:  whose name matches the pattern:      strlen
valgrind:  in an object with soname matching:   ld-linux-x86-64.so.2
valgrind:  was not found whilst processing
valgrind:  symbols from the object with soname: ld-linux-x86-64.so.2
valgrind:
valgrind:  Possible fixes: (1, short term): install glibc's debuginfo
valgrind:  package on this machine.  (2, longer term): ask the packagers
valgrind:  for your Linux distribution to please in future ship a non-
valgrind:  stripped ld.so (or whatever the dynamic linker .so is called)
valgrind:  that exports the above-named function using the standard
valgrind:  calling conventions for this platform.
valgrind:
valgrind:  Cannot continue -- exiting now.  Sorry.

The reason is a striped glibc. To work properly, Valgrind needs to overwrite some of the system functions the glibc provide. It does this by getting the symbols by name out of this library. This is of course not possible if all the symbol names are removed. You can prove this by executing nm /lib/ld-linux-x86-64.so.2. Gentoo provides a FEATURE=splitdebug, which adds debug libraries to the installation. Unfortunately setting this feature in /etc/make.conf, means setting it global. Gentoo is known as being configurable as no other distribution out there and of course we can set a feature for one program only. To do so, create a file called glibc in /etc/portage/env/sys-libs/ and add the following content to it.

FEATURES="splitdebug"

After a rebuild of glibc by executing emerge --oneshot glibc, we have a working Valgrind.

As all programs, Valgrind isn’t perfect. Version 3.5 shows many false/positive hits on my system, but fortunately the development goes on. Currently there is no newer version available in the Gentoo tree. Anyway it is not necessary to build one yourself, to get a more recent version. Using layman and the overlay tree of Flameeyes will let you integrate the development version of Valgrind seamlessly into your system. For a general How-to of layman check out this Users’ guide. In short, something like the following should be sufficient:

layman -a flameeyes-overlay
layman -s flameeyes-overlay
echo "=dev-util/valgrind-9999 **" >> /etc/portage/package.keywords
emerge valgrind

Installing the development version of Valgrind is optional of course.

Know your tools

One usage of Valgrind could be look like this:

valgrind --leak-check=full --leak-resolution=high ./VirtualBox

Beside other errors it also shows this error message on my system:

==27174==    at 0x4C26C09: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==27174==    by 0x4C26CB9: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==27174==    by 0xBA8967F: ??? (in /usr/lib64/libglib-2.0.so.0.2400.2)
==27174==    by 0xBA89E9D: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.2400.2)
==27174==    by 0xBA89F86: g_slice_alloc0 (in /usr/lib64/libglib-2.0.so.0.2400.2)
==27174==    by 0xC204847: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.2400.2)
==27174==    by 0xC1EB8A5: ??? (in /usr/lib64/libgobject-2.0.so.0.2400.2)
==27174==    by 0xC1ECE5D: g_object_newv (in /usr/lib64/libgobject-2.0.so.0.2400.2)
==27174==    by 0xC1ED494: g_object_new (in /usr/lib64/libgobject-2.0.so.0.2400.2)
==27174==    by 0x72A495F: ??? (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x72A0D4F: ??? (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x7289264: QGtkStyle::QGtkStyle() (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x7215DB6: QStyleFactory::create(QString const&) (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x6F5B7FC: QApplication::style() (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x6F61DFF: QApplicationPrivate::initialize() (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x6F61E88: QApplicationPrivate::construct(_XDisplay*, unsigned long, unsigned long) (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x6F61FF3: QApplication::QApplication(_XDisplay*, int&, char**, unsigned long, unsigned long, int) (in /usr/lib64/qt4/libQtGui.so.4.6.3)
==27174==    by 0x44BC38: TrustedMain (main.cpp:371)
==27174==    by 0x44C649: main (main.cpp:651)

If you analyze the backtrace, you see that something in libQtGui is leaking memory. I don’t want to blame someone for it or make a statement if this is right or wrong, I just want to get rid of it, to be able to easily spot errors VirtualBox itself produce. To do so, add --gen-suppressions=all to the Valgrind call. This will produce something similar like this:

{
 Memcheck:Leak
 fun:memalign
 fun:posix_memalign
 obj:/usr/lib64/libglib-2.0.so.0.2400.2
 fun:g_slice_alloc
 fun:g_slice_alloc0
 fun:g_type_create_instance
 obj:/usr/lib64/libgobject-2.0.so.0.2400.2
 fun:g_object_newv
 fun:g_object_new
 obj:/usr/lib64/qt4/libQtGui.so.4.6.3
 obj:/usr/lib64/qt4/libQtGui.so.4.6.3
 fun:_ZN9QGtkStyleC1Ev
 fun:_ZN13QStyleFactory6createERK7QString
 fun:_ZN12QApplication5styleEv
 fun:_ZN19QApplicationPrivate10initializeEv
 fun:_ZN19QApplicationPrivate9constructEP9_XDisplaymm
 fun:_ZN12QApplicationC1EP9_XDisplayRiPPcmmi
 fun:TrustedMain
 fun:main
}

To let Valgrind ignore this error in the future, copy the text into a file vbox.supp and start Valgrind with --suppressions=vbox.supp. Viola, this specific error isn’t shown anymore. The format used there is easy to understand and you can of course tweak this much more. E.g. you could replace some of the fun: entries by “...“. This is a placeholder for one or more functions calls with any name. Beside making suppression rules more general you can of course add as much as you like. Adding a name at the top make it easy to identify the different rules. For all the possibilities have a look at the documentation. Just for the curious, Valgrind is using such a file itself. Have a look at /usr/lib/valgrind/default.supp. You may also have noted that the function names in the normal error message differ from the one in the suppression list. The former is in the demangled form and the later in the saved form. You could force Valgrind to print mangled function names by adding the --demangle=no parameter to the call. This becomes handy if you manually create suppression lists.

Conclusion

By using suppression rules for the own application, unimportant errors could be eliminated in the output of Valgrind. With this is in mind there is no excuse anymore for memory leaks in the self developed applications. Beside memory leaks, Valgrind also finds places where uninitialized variables are in use or where memory is used which isn’t allocated by the application. Also these tests could be filtered out by suppression rules.

Changing the default behavior of built-in Cocoa controls

June 13, 2010Cocoa, ITArchive, Cocoa, delegate, Mac OS X, NSCell, NSControl, NSSearchField, objc, Objective-C, programming, search fieldcp

Apple has many task specific controls built into Cocoa. They are all well designed and have most of the functionality a user and a developer expect. One of this controls is the NSSearchField. This control has a special design which allows the user to recognize the provided functionality with ease. It is so well-known that Apple uses the design even on there website. It has support for menus (e.g. for recent search items), auto completion, a cancel button, and so one. Although this is mostly feature complete, there are sometimes cases where you like to extend it. In this post, I will show how to add another visual hint to this control when a search term isn’t found. The aim is to change the background of the underlying text edit to become light red to visual mark the failed search.

Understanding how Cocoa works

The NSSearchField class inherits from an NSControl. NSControls are responsible for the interaction with the user. This implies displaying the content in a NSView, reacting to user input like mouse or keyboard events and sending actions to other objects in the case the status of the control has changed. Usually a control delegate the first two tasks to a NSCell. The main reasons for this are to persist on good performance even if there are many cells of the same type (like in the case of a table) and to be able to exchange the behavior of the control easily (like in the case of a combobox which also allow typing in a text field). With this information in mind we know that we need to overwrite the drawing routine of the cell (NSSearchFieldCell) to achieve our goal. The implementation is straight forward and shown in the following:

@interface MySearchFieldCell: NSSearchFieldCell
{
  NSColor *m_pBGColor;
}
- (void)setBackgroundColor:(NSColor*)pBGColor;
@end

@implementation MySearchFieldCell
-(id)init
{
  if (self = [super init])
    m_pBGColor = Nil;
  return self;
}
- (void)setBackgroundColor:(NSColor*)pBGColor
{
  if (m_pBGColor != pBGColor)
  {
    [m_pBGColor release];
    m_pBGColor = [pBGColor retain];
  }
}
- (void)drawInteriorWithFrame:(NSRect)cellFrame inView:(NSView *)controlView
{
  if (m_pBGColor != Nil)
  {
    [m_pBGColor setFill];
    NSRect frame = cellFrame;
    double radius = MIN(NSWidth(frame), NSHeight(frame)) / 2.0;
    [[NSBezierPath bezierPathWithRoundedRect:frame
          xRadius:radius yRadius:radius] fill];
  }
  [super drawInteriorWithFrame:cellFrame inView:controlView];
}
@end

The user can set a custom background color by using setBackgroundColor. Also it is possible to reset the background color by passing Nil to this method. The method drawInteriorWithFrame draws a rounded rectangle on the background and forwards the call to the super class afterward.

Replacing the cell class of a control

The next step is tell the control to use our own cell class and not the default one. Although there is a setCell method defined in NSControl it is not as easy as one might think. Creating an instance of MySearchFieldCell and passing it to setCell after the NSSearchField is created will not have the expected effect. The reason for this comes from the fact that the cell is initialized when the control is created. This includes setting all properties and targets for the actions. If one replaces the cell afterward these setting will be get lost. Later on, I will show a method how to keep this configuration, but for now we will start with an easier approach.

When the control creates a cell object it asks a static method for the class name to use. This method is called cellClass. If we overriding this method with our own one, we are able to return MySearchFieldCell. The following code demonstrates this:

@interface MySearchField: NSSearchField
{}
@end

@implementation MySearchField
+ (Class)cellClass
{
  return [MySearchFieldCell class];
}
@end

Now, if you use MySearchField instead of NSSearchField when creating search fields, you are done. Unfortunately this isn’t always possible. First you may not be able to inherit from NSSearchField for whatever reason and second this will not work when you are use the Interface Builder (IB) from Xcode. There you can’t easily use your own version of NSSearchField, but you have to stick with the original one. Before we proceed the obligatory screenshot:

The power of Archives

What we need is a method of setting our own cell class even when the control is already instantiated. In Cocoa it is possible to Archive and Serialize any object which implements the NSCoding protocol. Archiving means that the whole class hierarchy, with all properties and connections, is saved into a stream. Xcode makes heavy use of this in the nib file format where all the project data of the IB is written in. This alone doesn’t help us much, but additional to the archiving and unarchiving work, it is possible to replace classes in the decoding step. The relevant classes are NSKeyedArchiver and NSKeyedUnarchiver. NSKeyedUnarchiver has a method setClass:forClassName which allow this inline replacement and the following code shows how to use it:

NSSearchField *pSearch = [[NSSearchField alloc] init];
/* Replace the cell class used for the NSSearchField */
[NSKeyedArchiver setClassName:@"MySearchFieldCell"
      forClass:[NSSearchFieldCell class]];
[pSearch setCell:[NSKeyedUnarchiver
      unarchiveObjectWithData: [NSKeyedArchiver
        archivedDataWithRootObject:[pSearch cell]]]];
/* Get the original behavior back */
[NSKeyedArchiver setClassName:@"NSSearchFieldCell"
      forClass:[NSSearchFieldCell class]];

Basically this creates an archive of the current NSSearchFieldCell of the NSSearchField, which is instantly unarchived, but with the difference that the NSSearchFieldCell class is replaced by MySearchFieldCell. Because the archiving preserve all settings the new created class will have the same settings like the old one. The last call to NSKeyedArchiver will restore the default behavior.

Conclusion

This post should have removed some of the mysteries of the control and cell relationship in Cocoa. Additional to a simple derivation approach, a much more advanced way for setting the cell of a control was shown. This allows the replacement of any class hierarchy without loosing any runtime settings. If you need such replacements much more often or working with the IB, you should have a look at Mike’s post which shows a more generic way of the archive/unarchive trick.

Installing Linux on a USB hard disk for the MacBook Pro

April 9, 2010IT, Rootadministration, boot, Boot Camp, EFI, FireWire, Gentoo, hard disk, installation, Linux, Mac OS X, MacBook Pro, removable media, Ubuntu, USBcp

One of the features of Mac OS X I love, is the possibility to install Mac OS X on any attached removable media, like a FireWire or USB hard disk. This makes it really easy for me to test VirtualBox on the several versions of Mac OS X we support (formerly Tiger, now Leopard and Snow Leopard). The advantage of this setup is that I don’t waste disk space for operation systems I usually don’t use very often. Currently I have a 150GB hard disk in my MacBook Pro which is really not that much if you deal in the virtualization business. There are several test VM’s of any kind of guest operation systems and of course the ISO’s to install them. The second main OS, I do much of my work, is Linux. For this I have a standard PC with Gentoo on it, which have all that I need. Unfortunately this doesn’t really help when I on travel. As I soon be away for some time, I decided I need, at least for testing, the same flexibility mentioned above for a Linux installation. And here the problems start to arise. Of course Apple didn’t really support installing other OS’s than Mac OS X on Apple hardware. Yes, there is Boot Camp, but this is mainly for Windows, is very inflexible and doesn’t really help if you try to install something on another place than the integrated hard disk. There are projects like rEFIt, which even makes Boot Camp superfluous, but this project has really bad USB boot support. In the following I will explain how it is even possible to install Ubuntu 9.10 on a USB hard disk.

The hardware used, as already said, is a MacBook Pro 3,1 and a Western Digital My Passport Essential 500GB USB hard disk. Other combination may work, but I don’t guarantee this, as always. Also you should be warned that anything I describe here could destroy your existing installation and I’m not responsible for that. Doing some kind of backup might be a good idea. Time Machine is easy to use ;).

Before we start, as usual, the credits for some sites I get my information from. This is mainly the Produnis blog, the Blog of Chris, the Ubuntu wiki and of course the guys around the Grub2 development.

Creating the EFI boot loader

Apple doesn’t use the legacy BIOS to boot their machines, they use the Extensible Firmware Interface (EFI). This new way of booting operation systems is very flexible, as the name suggest, but has several drawbacks, like most of the standard operation system doesn’t speaks it language. Although Linux can be configured to use EFI directly we will emulate a legacy BIOS in the following. For this we need a connector which makes EFI and Linux happy and let them both work smoothly together. This connector is Grub2, which is in development for several years now. It’s the successor of Grub and is the standard in many popular Linux distributions these days. We have to build a version our self, for which an existing Linux installation is really helpful. I used my 64-bit Gentoo installation. First you have to find out if the EFI installation on your target Apple machine is 64 or 32-bit. You can do this by executing:

ioreg -l -p IODeviceTree | grep firmware-abi

This will return EFI64 or EFI32 respectively. In my case I need the 64-bit version, which is a little bit surprising when I consider that my MacBook Pro isn’t able to boot a 64-bit Snow Leopard. Anyway, grab the latest version of Grub2 and unpack it on the Linux machine. Please note that you need a gcc with multilib support if you are targeting an architecture which isn’t the same as the host one. Use the following to configure Grub2 and to build it. Of course you have to adjust the target architecture if it is a different one.

./configure --with-platform=efi --target=x86_64 --disable-werror
make

When this is finished you create the EFI package by executing

./grub-mkimage -d . -o bootx86.efi *.mod

Here I included all modules which are available. If size matter for you, you could of course make a selective choice on the modules included. I didn’t test this myself, so you have to find out yourself which one are important.

Whipping the USB hard disk into shape

Next we have to prepare the USB hard disk for the new installation. In the following I assume your USB hard disk doesn’t contain any valid data and could be reformatted without data lose. Make a backup of your data first if this isn’t the case on your side. Apple uses the GUID Partition Table scheme to organize their partitions on a hard disk. This specification is part of EFI and remove many limitations of the Master boot record (MBR) scheme, which is widely used in the PC world. That is e.g. the disk size limitation of 2TB or the maximum of 4 primary partitions. You reformat your disk, using the Disk Utility application of Mac OS X. Make sure all existing partitions on the disk are unmounted. When necessary, change the partition scheme from MBR to GUID in the Options dialog of the Partitions window. Select the partitions count you want to use. You need at least 3 partitions to make Linux works fine. My partition scheme looks like in the following: As you can see I have 5 partitions configured. The first one is an additional Snow Leopard installation for testing. I also added a Data partition at the end for making the data transfer between the different operation systems as easy as possibly. LINUXBOOT is a small partition which will contain the EFI boot loader (size it 50MB or something like that). Linux Swap, obviously, will become the swap partition of the Linux installation. DISK1S5 is the Linux root partition itself. The Data partition has to be formated as Mac OS Extended. Don’t use the Journaled version of HFS+, cause this makes trouble on the Linux side. The other partitions have to be formatted as MS-DOS (FAT).

After applying the changes we can add the EFI boot loader to the LINUXBOOT partition. The Apple EFI implementation is searching for a file with the efi extensions on all bootable hard disks. Mount LINUXBOOT and create a efi/boot directory on the root path. Copy the bootx86.efi file into the boot/ directory. As bootx86.efi is a Grub2 boot loader we need a valid Grub2 configuration file. The following grub.cfg shows the configuration for a Ubuntu 9.10 i386 installation. For the 64-bit version or any other version of Ubuntu the settings might be slightly different.

menuviewer="text"
timeout=10
default=0
set F1=ctrl-x
menuentry "ubuntu-9.10-desktop-i386"
{
 fakebios
 search --set -f /boot/vmlinuz-2.6.31-14-generic
 linux /boot/vmlinuz-2.6.31-14-generic root=UUID=4e140981-4ab3-41a2-a2fb-26b1287beb87 ro quiet splash noefi video=efifb
 initrd /boot/initrd.img-2.6.31-14-generic
}
menuentry "ubuntu-9.10-desktop-i386 single"
{
 fakebios
 search --set -f /boot/vmlinuz-2.6.31-14-generic
 linux /boot/vmlinuz-2.6.31-14-generic root=UUID=4e140981-4ab3-41a2-a2fb-26b1287beb87 ro noefi video=efifb single
 initrd /boot/initrd.img-2.6.31-14-generic
}
menuentry "ubuntu-9.10-desktop-i386 text"
{
 fakebios
 search --set -f /boot/vmlinuz-2.6.31-14-generic
 linux /boot/vmlinuz-2.6.31-14-generic root=UUID=4e140981-4ab3-41a2-a2fb-26b1287beb87 ro noefi vga=normal
 initrd /boot/initrd.img-2.6.31-14-generic
}
menuentry "Mac OS X"
{
 search --set -f /usr/standalone/i386/boot.efi
 chainloader /usr/standalone/i386/boot.efi
}
menuentry "CD"
{
 appleloader CD
}
menuentry "mbr"
{
 appleloader HD
}
menuentry "reboot"
{
 reboot
}

You have to change the root UUID to the one the Ubuntu installer will assign to your hard disk after installation. Just check the fstab file when the installation has finished. The first entry boots Linux with a splash image enabled. The second one is for the single user mode in the case something went wrong. Please note the video=efifb option, which enables the graphical mode in the boot phase.

Installing Ubuntu

Most of the installation process is straight forward and doesn’t need any special attention. Download the version of your choice from one of the mirrors, burn it on CD and start the installation. You can select the CD as boot medium by pressing Alt when your Mac starts. When the installer ask for the partition scheme, you have to switch to “manual choice”. Select the DISK1S5 (your what it is in your case) as the root / partition and change the filesystem type to ext3. Also remember the path to the system partition, cause you will need it later again. Select the swap partition and change its type to swap. Proceed with the rest of the installation until the last dialog. There select “advanced settings” and change the boot loader target from hd0 to /dev/sdXX, where you replace XX to the path you used previously in the partition tool.

If all went right you should be able to select the LINUXBOOT partition by pressing Alt when your Mac starts. After that Grub2 should shows up, you will be able to boot into your freshly installed Ubuntu.

Conclusion

In this post I showed how to easily add the possibility to boot Linux on your MacBook Pro. With the external USB hard disk solution, no internal valuable space is wasted. Of course the speed isn’t the same as if the OS would be installed on the internal drive, but for testing software on different operation systems this is satisfactory. To increase the speed a little bit more, an external FireWire hard disk could be used.

Why an ending slash on a URL matters (or not)

March 5, 2010IT, WebBlog, Google, PHP, search engine optimization, SEO, web crawler, web standards, WordPresscp

Recently I read a report from Google about the status of their own Web sites in terms of Search Engine Optimization (SEO). Uhh, Google, the search engine giant, is looking how the own sites are behaving when a Web crawler is stumble over them! To be honest, it’s fully clear to me that in such a big company, like Google, not all is perfect in the sense of cooperative appearance. There are different projects, different teams, with different strengths and weaknesses, different priorities and of course different managers. To summaries the report: In some areas Google does a good job, but in most it doesn’t.

How to optimize

Although SEO even lead to a business case for some companies, I believe it’s not so hard to do and in my humble opinion the most important points, a Web site owner should take care of, are the following 6:

Usage of Web standards like XHTML 1.0 Transitional and making sure the Web site conforms to them.
Usage of the <title> tag.
Usage of meta tags like description and keywords.
Usage of the header tags <h1>, <h2>, and so one.
Adding canonical URL information to every site of the Web site, if a specific site is reachable from more than one URL.
Writing good content.

Although 6. is quite obviously, it’s the most important point and often people forget about it and wonder why there is no traffic on there Web site at all.

1.-5. are technical aspects and if a Web site owner is using e.g. WordPress the Web site should be in good shape, already. Of course this depends a little bit on the used theme and the plug-ins the user has installed. For item 1. I always propose to make a bug report if a theme or a plug-in doesn’t confirm to them, as I have done for the theme used in this blog. Conformity could be easily tested with the W3C Valitator.

Why one should use HTML tags like the title, the meta and the heading tag is also easy to understand. A Web crawler isn’t a human, so he can’t distinguish between structural information because e.g. the font size is different. Helping him by semantic marking some of the text with the available tags of HTML is therefor a good idea. For the same reason using HTML tables to layout a page is a bad idea. Although this was standard in Netscape 4.0 times it isn’t necessary anymore these days.

Slash or no slash

Item 5. is about giving the Web crawler a clear idea about the structure of your Web site in a whole. Comparing 64k-tec.de/test and 64k-tec.de/test/ as a human doesn’t seem to be very different. From a technical point of view, it is. Considering that the Web itself is grown up in a UNIX environment, the former points to a filename and the later to a directory. This means for a Web crawler two different sites are targeted. The easiest way to fix this, is to decide about the nomenclature globally used. Either use the one form or the other. WordPress uses the “ending with a slash” variant (most of the time). By the way, this is also important for 64k-tec.de/test/index.html and other variants. Another way is to tell the Web crawler the canonical address even if the site is served by another URL. This could be done by adding the link rel tag canonical to the header of the page. On my homepage this looks as follow:

As you see, even on the top-level domain a slash is added. WordPress does this automatically for you since version 2.3. On older versions plug-ins for this task are available. The canonical tag is a good way to make clear which address is the base URL of a specific page. On the other side I see some potential for improvements. I have found two places in my blog where the base address isn’t targeted right. The first one is the tag for the site index relationship. It’s noted as follow, on my blog:

The second one is the link of the logo presented on top of every page on the blog. It use the following link:

64k

As you see the ending slash is missing, both times. It is not really a problem, cause the page itself use the canonical tag. The second mistake is clearly a failure of the theme. It’s not fully clear to me if the first wrong target is a failure of the theme or WordPress itself. It also happens with the default WordPress theme (version checked is 2.9.2).

Conclusion

Creating a Web site which is easily understandable by a Web crawler isn’t any magic. Of course you could make a pure science out of it. There are tons of plug-ins for WordPress available. On the other side considering some simple rules will help a lot. Reading Google’s hints or using Google webmaster tools might help, too. Even for Web crawlers from other companies.

On a last note, here is a nice article about how to effectively keep users out of your blog ;).

FRITZ!Box tuning part 3: Using the VoIP phone line from everywhere

February 6, 2010IT, Rootcommunication, FritzBox, Mac OS X, phone calls, remote access, softphone, VoIPcp

The contract with my Internet service provider includes a VoIP connection. Together with the FRITZ!Box 7270 all phone calls are done over the Internet, an additional conventional phone connection isn’t necessary anymore. One of the benefits of this is that you can connect to your registrar from everywhere in the world. But that’s theory, cause e.g. my provider doesn’t allow a connection if you are not in the network of the provider itself. Of course there are free services like sipgate or even Skype. There you could make free calls within the services itself, but as soon as you like to call a real phone number you have to pay. They are cheap, no question, but my contract includes a flat rate within Germany. That’s even cheaper. So what I like to do is to use my VoIP phone connection even when I’m not at home. In the following third part of the FRITZ!Box tuning series, I will explain how to achieve this. As already said in the first two post, where you at least should read the first one, I’m not responsible for anything happens to your FRITZ!Box after you have tried what is described here.

Adding a softphone to the FRITZ!Box

The FRITZ!Box fully supports softphones in its basic configuration. Adding new softphones is done in the web frontend in the extended configuration section. There is a wizard for this, where you have to answer some question about your new device. Select phone as device, LAN/WLAN (IP-Phone) for the connection type, choose a name and a password. You get a new internal phone number which usually starts at 620 for the first created device. The FRITZ!Box try’s than to connect to the new device. You can skip this for now. After that you have to select which official phone number the new device should use. This is important cause this will be displayed as the caller id when you make a call. Now you could choose if the new softphone should react on all incoming calls, regardless of the phone number called, or only on a specific one.

Configuring the client software

I will explain the client setup with the help of a free VoIP software called Telephone. It’s only available for Mac OS X, but there are many other VoIP clients out there which also works for other operation systems. After downloading and installing it you have to create a new account. Select a descriptive name for the account. For the domain use fritz.box, as username you have to use the internal number the FRITZ!Box selected above. Lets use 620 for now. The password is the one you chose previously. After the account is created, I had to select Substitute “00” for “+” in the advanced settings, cause the phone numbers in my address book are all saved with the international phone prefix of Germany which is +49. Assuming you are working in your home network right now, you should be able to connect with Telephone to your FRITZ!Box. Some simple tests like making an outgoing and incoming call with an additional mobile phone should verify a working setup. Some nice features like the Mac OS X Address Book integration, which also transfers incoming caller id’s to real names if they are found in the address book, making Telephone a really useful application. For further phone call management functionality on Mac OS X, I can advice you to take a look at the also free software Frizzix.

Allowing calls from everywhere

All the previously isn’t any magic, cause it uses build in support of the FRITZ!Box. But this article is about allowing connections to the VoIP part of the FRITZ!Box from everywhere. To make this happen we have to edit an internal configuration file of the FRITZ!Box. This time we aren’t change the ar7.cfg file, but a file called /var/flash/voip.cfg. As the name suggest there is most of the VoIP configuration included. Use nvi to edit it and search for a section called extensions. There should be a newly created one which have a value extension_number with 620. Change the value reg_from_outside from no to yes. This section should then look like the following:

extensions {
    enabled = yes;
    username = "$$$$SSFSDFSOPKSFDOPK;LWE§REWSDFMKFSLDF3232SDFSDFSDF";
    authname = "";
    passwd = "$$$$DFS342ASDFSDFDSFDS§344WLKKHMSJHAJHASDAHQASLKADJSA";
    extension_number = 620;
    reg_from_outside = yes;
    tx_packetsize_in_ms = 0;
}

Save the file and reboot your FRITZ!Box by typing reboot. Next we have to tell Telephone to use a proxy when connecting to the FRITZ!Box. Go to the advanced settings and add your DynDNS name, in this example it would be xtestx.dyndns.org, to the proxy field. That’s all. Now you should be able to make calls from every network you are currently logged in.

Conclusion

This simple change to the internal configuration of the FRITZ!Box allows you to use your VoIP account from everywhere over the world. As nice this feature is, I like to add some words of caution at the end. First you need a good download and especially a good upload connection speed in your home setup, cause all phone calls are routed over the FRITZ!Box. Secondly I didn’t know if VoIP transfers are encrypted in any way, so be aware that there is the possibility someone monitor your calls. And as last note you should understand that you opened a port on your FRITZ!Box for everyone, which in the case your password is stolen, could be abused. In the worst case someone use your account to SPAM other people or call expensive service numbers.

FRITZ!Box tuning part 2: Access your home network with OpenVPN

February 2, 2010IT, Rootadministration, communication, FritzBox, Gentoo, Linux, OpenVPN, PKI, remote access, VPNcp

AVM has built a VPN server into the FRITZ!Box, why should I use some other software for this job, you may ask. The reason is quite simple: the build in one is a piece of closed source software written by AVM and there is only one official client which could be used to connect to it, the FRITZ!VPN software. This client software is only available for the Windows operation system family and so by no means anything useful to me. I’m pretty sure they are using some official protocol like IPSec, so it might be possible to connect to the FRITZ!Box with other clients as well, but that’s something I didn’t want to try. OpenVPN on the other side is a rock solid open source software which could be used from many popular OS’s these days. Even graphical clients, like TunnelBlick for Mac OS X, are available. So here comes the second article of the FRITZ!Box tuning series, which will explain how to convert your FRITZ!Box into a OpenVPN server, where any number of clients can concurrently be connected. I highly recommend to read the first part of this series, because this post is build on top of the stuff done there. This count especially for the filesystem layout on the usbstick and the way additional software is started. Also in the following it will be helpful to have ssh access to the FRITZ!Box all the time. As already written in the first part, there is no guarantee that the information presented here will work on your side or that I’m responsible for anything happen to your FRITZ!Box. In preparation of the following you need access to a second OpenVPN installation which will be used to create all necessary certificates and keys and which could be used to test the installation afterward. I’m using a Gentoo Linux host where you could install OpenVPN simply by executing emerge openvpn. Make sure you have the examples USE flag set to get all the helper scripts which make the life much more easier.

The agony of choice

OpenVPN could be configured in many different ways. So first of all we have to decide which features of OpenVPN we will use. In its simplest variant OpenVPN could be used with one preshared static key. The advantage of this setup type is the easy configuration. The disadvantage is that only one client could be connected at the time. That is not what we want, so we will configure our setup to be using a Public Key Infrastructure (PKI). This allows more than one client to be connected at the time and it is possible to mark single keys invalid without affecting other client keys. The second question which has to be answered is, if we want use routing or bridging in our setup. Again, routing is simpler to set up and also a little bit faster than bridging. Bridging on the other side allows the tunneling of non IP protocols such as IPX, tunnel all the IP broadcast and make it possible to use the same subnet on the client side as on the server-side. Especially the last point is a nice feature as with routing all hosts on the server-side needs a routing table entry to find the connected clients. But my main reason for preferring bridging over routing is the broadcast feature of the first one. In my home network works a NAS station as a file server. This includes of course simple filesystem sharing with AFP and NFS, but also serves this box audio data with the Digital Audio Access Protocol (DAAP). DAAP is used by Apple in iTunes to share music with other clients in the network and that is what my NAS station does. This protocol, as well AFP, using broadcast messages (ZeroConf) to find some potential clients. So now it should be clear why I want bridging, it allows me to hear music or to simply connect to my file station without any interaction.

Building up a PKI

To create all necessary certificates and keys we are using the easy-rsa scripts from OpenVPN. They are located under /usr/share/openvpn/easy-rsa. First we change the vars file. On the bottom of the file are personal information defined which should be changed to fit to your environment. The entries looking as follow on my side (with a pseudo email address).

export KEY_COUNTRY="DE"
export KEY_PROVINCE="Sachsen"
export KEY_CITY="Dresden"
export KEY_ORG="64k"
export KEY_EMAIL="you@yourdomain.org"

You could also increase the KEY_SIZE from 1024 to 2048 within that file. This should make your keys unbreakable for the next few decades years. Now we are ready to create the master-key and the master certificate of the Certificate Authority (CA) by executing:

source vars
./clean-all
./build-ca

Please be aware that this will delete all previously created keys. Answering all questions with the default values should be sufficient. Next the key for the server has to be created.

./build-key-server server

You can again accept all default values. A challenge password is not necessary, but you have to sign the certificate. Now its time for our first client key and certificate. You could the following repeat again and again, for every new client you want allow to access the OpenVPN server.

./build-key client

Again, don’t forget to sign the key. If you ever have to mark a client certificate invalid you could do it with the following command:

./revoke-full client

This will create or update a Certificate Revocation List (CRL) file which will be later used by the server to verify the certificate integrity of a connecting client. For now we create an empty one with

KEY_CN="" KEY_OU="" KEY_NAME="" $OPENSSL ca -gencrl -out keys/crl.pem -config $KEY_CONFIG

The last step in this key building marathon is to create a Diffie-Hellman key. This key will be used to initiate the secure channel between the two parties. Its created by:

./build-dh

Creating the server and client configuration

We start with the configuration of the server:

# set the path to the tap device
dev tap
dev-node /var/tmp/tap0
mssfix
tun-mtu 1500

# TCP or UDP server?
proto udp
port 1194

# CA and key files
ca /var/media/ftp/FLASH-DISK-01/addons/openvpn/keys/ca.crt
cert /var/media/ftp/FLASH-DISK-01/addons/openvpn/keys/server.crt
key /var/media/ftp/FLASH-DISK-01/addons/openvpn/keys/server.key
crl-verify /keys/crl.pem # this is opened after the chroot

# Diffie hellman parameters
dh /var/media/ftp/FLASH-DISK-01/addons/openvpn/keys/dh1024.pem

# bridge setup
server-bridge 192.168.220.1 255.255.255.0 192.168.220.50 192.168.220.99
mode server
tls-server
push "dhcp-option DNS 192.168.220.1"

ifconfig-pool-persist ipp.txt
client-to-client

# compression?
comp-lzo

# The maximum number of concurrently connected clients we want to
# allow.
max-clients 20

# It's a good idea to reduce the OpenVPN daemon's privileges after
# initialization.
user nobody
group nobody

# Put openvpn in a jail.
chroot /var/media/ftp/FLASH-DISK-01/addons/openvpn

# The persist options will try to avoid accessing certain resources on
# restart that may no longer be accessible because of the privilege
# downgrade.
persist-key
persist-tun

# where to log
log-append /var/media/ftp/FLASH-DISK-01/addons/openvpn/log/openvpn.log

# verbose level for debugging
;verb 4

# make sure the connection is kept alive
keepalive 10 120

For bridged mode it is essential that OpenVPN uses a TAP device and not the TUN device mode (see here for the reason). We use UDP as the transport protocol and enable compression. Also we want that OpenVPN drops its privileges after the initialization and chroot into a jail to make it a little bit more secure. As the subnet for clients 192.168.220.0/24 is used. That is the same as my internal network uses. OpenVPN is allowed to use addresses ranging from 192.168.220.50 to 192.168.220.99. You have to make sure that the FRITZ!Box (or any other DHCP server in your network) doesn’t serve addresses from that range. Of course you could change some of the settings to your needs, but then you have to make sure they match the configuration of the client.

Next we create the client configuration, which looks as follow:

# Specify that we are a client and that we will be pulling certain
# config file directives from the server.
client

# The hostname/IP and port of the server.
remote xtestx.dyndns.org 1194

proto udp
dev tap

pull

mssfix
tun-mtu 1500

# SSL/TLS parms.
tls-client
ca keys/ca.crt
cert keys/client.crt
key keys/client.key

ns-cert-type server

# Keep trying indefinitely to resolve the host name of the OpenVPN
# server. Very useful on machines which are not permanently connected
# to the Internet such as laptops.
resolv-retry infinite

# Don't enable this; It must be disabled for iTunes to find the iTunes
# server and for AFP broadcast in general.
#nobind

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nobody

# Try to preserve some state across restarts.
persist-key
persist-tun

# Enable compression on the VPN link.
comp-lzo

# debug level
;verb 4

mute-replay-warnings
mute 20

You have to change the remote hostname to your needs. You may also add the full path to the certificates and keys depending on your client OpenVPN installation.

Next we pack all files up for the server and the client respectively. The server needs the following one:

server.conf
keys/ca.crt
keys/crl.pem
keys/dh1024.pem
keys/server.crt
keys/server.key

The client needs this one:

client.conf
keys/ca.crt
keys/client.crt
keys/client.key
keys/server.crt

Customizing the FRITZ!Box

Connect to your FRITZ!Box and copy all the server files to the usbstick. I created a directory openvpn under the /var/media/ftp/FLASH-DISK-01/addons path. Next we need the OpenVPN binary for the FRITZ!Box. You can get it here. Unpack the file and copy the openvpn binary to /var/media/ftp/FLASH-DISK-01/addons/bin on the FRITZ!Box. Make sure the file mode has the executable bit set for the user. All files are now on the right places, so we can adjust our start script. Add the following code to the startup.sh file.

# Add a nobody user
echo 'nobody:x:65534:65534:nobody:/:/bin/false' >> /var/tmp/passwd

# Add some groups
echo 'root:x:0:' > /var/tmp/group
echo 'nobody:x:65534:' >> /var/tmp/group

# Create a tap device for openvpn
mknod /var/tmp/tap0 c 10 200

# Start openvpn
${BASE}/openvpn/bin/openvpn --config ${BASE}/openvpn/server.conf --daemon

As you see, we add a new nobody user to the Linux system. Then the device node for the TAP device is created and at the end the OpenVPN server is started. As we using the bridged setup, we have to add the new device to a bridge. Fortunately we haven’t to create one, cause the FRITZ!Box itself is working in bridged mode. You can confirm this by executing:

brctl show lan

To let the system automatically add our TAP device to the lan bridge, we change the internal FRITZ!Box configuration. Edit the /var/flush/ar7.cfg file with nvi and search for the string brinterfaces. There should be one section with the name lan. Simply add the new tap0 interface to the interfaces value. The section should then look like this:

brinterfaces {
     name = "lan";
     dhcp = no;
     ipaddr = 192.168.220.1;
     netmask = 255.255.255.0;
     dstipaddr = 0.0.0.0;
     interfaces = "eth0", "ath0", "tap0", "wdsup1", "wdsdw1",
                  "wdsdw2", "wdsdw3", "wdsdw4";
     dhcpenabled = yes;
     dhcpstart = 192.168.220.100;
     dhcpend = 192.168.220.200;
}

Now, as soon the tap0 interface is created it will be added to the bridge. The last important point in the server setup is to change the internal firewall to allow connections to port 1194 from the outside. Search for forwardrules in the ar7.cfg file and add this new rule:

forwardrules =
               "tcp 0.0.0.0:7777 0.0.0.0:22 0 # SSH-Server",
               "udp 0.0.0.0:1194 0.0.0.0:1194 0 # VPN-Server";

Thats all for the server. A reboot of the FRITZ!Box should start the new OpenVPN server. If you have trouble you should manually start the OpenVPN server and increase the debugging level, as shown in the above configuration with the keyword verb.

Setting up the client

On the client side we need the files we packed above. On Gentoo the OpenVPN configuration is located at /etc/openvpn. Copy the configuration, the certificates and the key file to that place. The OpenVPN start script of Gentoo looks for the configuration file, based on the start script name. So we have to create a link like this:

cd /etc/openvpn
ln -s client.conf openvpn.conf

By the way, this lets you configure more than one OpenVPN connection at the time. Just create a link in /etc/init.d with the new configuration name and link it to the openvpn start script.

To make a first attempt to start the OpenVPN connection type

/etc/init.d/openvpn start

If all goes well you should have a new device tap0 when you execute ifconfig. Also there should be a route set, which point to the tap0 device when the 192.168.220.0/24 subnet is the target. Of course now you should be able to connect to any host in your home network by the IP. I didn’t experiment enough with the pushing of the name server from the OpenVPN server, instead I just added the FRITZ!Box as an additionally name server to my /etc/resolv.conf. So here is room for further experiments.

To let the OpenVPN client start at boot time add it to the default runlevel by executing:

rc-update add openvpn default

Also note that with the above configuration the client always try to reconnect to the server, even if the FRITZ!Box is rebooted, which is a nice feature.

Conclusion

This setup allows you to connect to your home network in a very secure way without loosing any functionality. You are able to browse network shares on the local network or using services like DAAP without any further interaction. The public key infrastructure make this setup a candidate for small business installations. If an employee leave the company his certificate is simply revoked without affecting the access of other employees.

http://wwww.cswpro.de