OpenVZ is an OS-level virtualization technology. This means the host OS is partitioned into compartments/containers with resources assigned to each instance nested within.
In OpenVZ there are two types of resources, dedicated and burst. A dedicated resource is one where the VPS is guaranteed to get such if requested; these are "yours". Burst resources come from the remaining unused capacity of the system. The system may allow one VPS to borrow resources like RAM from another VPS when the second one is not using them. Since it is borrowing, such resources have to be returned as soon as possible. Should the other VPS want their dedicated resources back, your processes might become unstable or terminated.
Since OpenVZ is an OS level virtualization, It consumes far less resources per VPS container than a full virtual environment. On two hosts with identical hardware and subscription rates, OpenVZ should perform better than KVM because it doesn't do full emulation. For example, it doesn't need to run multiple full OS kernels, as it can share the single kernel between multiple VPSes. The result is significant memory and CPU savings. In fact, most of the kernel memory usage is not charged to the VPS at all, instead it is only charged what each particular VPS needs in addition to the main kernel.
KVM is a hardware virtualization technology. This means the main OS on the server simulates hardware for another OS to run on top of it. It also acts as a hypervisor, managing and fairly distributing the shared resources like disk and network IO and CPU time.
KVM does not have burst resources; they are all dedicated or shared. This means resources like RAM and diskspace are usually much harder to overcommit without endangering all user data. The downside with KVM is that if the limits are hit, the VPS must either swap, incurring a major performance penalty, or start killing its processes. Unlike OpenVZ, KVM VPSes cannot get a temporary reprieve by borrowing from it's peers as their dedicated resources are completely isolated.
Because KVM simulates hardware, you can run whatever kernel you like on it (within limits). This means KVM is not limited to the Linux kernel that is installed in the root node. KVM can also run other x86 operating systems like BSD and Microsoft Windows. Having a fully independent kernel means the VPS can make kernel modifications or load its own modules. This may be important because there are some more obscure features that OpenVZ does not support.
Both OpenVZ and KVM are mature technologies with advantages and disadvantages to each. Selecting the appropriate technology may save you future headaches. To that end, please review the following list to see where your use fits.
Low Cost ($)