I don't yet have Compressor 2, but do have the manual. I imagine that you need to be looking for Dolby Digital file types which are, in fact, AC3 format. I don't think it refers to AC3 directly in the dialog boxes within Compressor, but I could be wrong.
Have a look at the screen shot on page 94 of the manual for what I am talking about. It appears that in the property inspector window there is an option for DD Pro settings in the encoder pane. Maybe this will point you in the right direction...