Ticking that box takes FSX out of its normal DirectX 9 mode and switches it to a DirectX 10 display engine.

DirectX 10 (DX10) support was promised for FSX during development. In the end however only a partial implementation was managed and they call this "DX10 Preview" mode. This only uses some DX10 features and is really only seen now as a demo of the sort of things they will be putting in the next version of FS

DX10 is a better and faster graphics engine than the previous DX9 but requires graphics cards that are designed to support it. It also (for reasons only known to MS) will only run on Vista. So to use DX10 in FSX you need a DX10 card and Vista. You will have this as the selection box only appears if you have a suitable system.

As you have seen DX10 makes for faster performance. However in FSX it is much more sensitive to correct compatibility in the objects designed for it. Aircraft and sceneries that display perfectly in DX9 may have problems in DX10 if they were not created with the correct FSX tools and in the correct format. Many aircraft are technically not fully FSX compatible in their design and creation and while they may work without visual problems in DX9 they "break" under DX10.

Some 3rd party suppliers offer updated "DX10 compatible" versions of their products (or patches to fix the existing ones) but most people have resigned themselves to some things not working properly and only using DX10 mode occasionally and with aircraft they know to work.