This kinda depends on what year.version.etc of G5 you got. Early one's had AGP video cards and simple PCI slots, much like the G4's.
Most G5's had 8x AGP slots where G4's could only have 2x and 4x AGP slots.
The 9800 is a 2x4x that will not work in a G5. The 9600 is a 4x 8x that works in the very last G4 (Dual Mirrored door) and earliest G5 4x8x.
There's not much more you can do for an AGP G5 since the ATI x800XT card is no longer made, the 9600 Pro is THE CARD!