.NET - What are Reference Assemblies?

What is the purpose of the Reference Assemblies directory under Program Files?

Reference Assemblies constitutes the published API for a specific version of the framework.

When you target a specific version of the framework in Visual Studio, the compiler uses type information from Reference Assemblies, and not from assemblies in the GAC. Not yet. Not until runtime.

Reference Assemblies contain no implementation, only metadata.

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v3.5
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5.1
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5.2
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.2
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.X

How do these assemblies compare to those in C:\Windows\Microsoft.NET\Framework\v4.0.30319 ?
How are they used?

Let's look at mscorlib.dll.

The Reference Assembly is smaller. It contains no implemetation, only metadata.

The Reference Assembly is older. It represents the assembly as it appeared when it was originally released, without any patches.

Since the Reference Assembly contains only metadata, the only patches that we are interested in, are changes to the API. Methods that were added. Interfaces that were changed.
When you build against a Reference Assembly you are targeting a version of the framework as it appeared, when it was publicly released, as opposed to the framework in your machine's GAC.

The concept is similar to C++ Include Files.

When you #include <windows.h>, the compiler permits you to call only those methods that are present in that version of Windows. If you are compiling for Windows 8, you don't want to use an API that was added for Windows 10.