This time around, I forgot I needed to redistribute from IPv4 Unicast BGP into IGP. This is required because when the VPNv4 BGP peers recurse to IGP (putting the IGP label on top of the VPN label), it has to know of the loopback in IGP. Even if it knows it by IPv4 BGP, that's not an IGP so it still won't work (meaning you may see the routes appear on the CEs but pings will fail, because the labeling is failing).
Other things to note, MP-eBGP multi hop and IPv4 Send-label are required.
IPv4 BGP Send-label to get labels across the Inter AS connection.