AssemblyVersion pretty much stays internal to .NET, while AssemblyFileVersion is what Windows sees. If you go to the properties of an assembly sitting in a directory and switch to the version tab, the AssemblyFileVersion is what you'll see up top. If you sort files by version, this is what's used by Explorer.
The AssemblyInformationalVersion maps to the "Product Version" and is meant to be purely "human-used".
AssemblyVersion is certainly the most important, but I wouldn't skip AssemblyFileVersion, either. If you don't provide AssemblyInformationalVersion, the compiler adds it for you by stripping off the "revision" piece of your version number and leaving the major.minor.build.