Unfortunately it appears that the port I missed was not on the carburetor. As if it were never drilled and used from the factory. The best I can tell from looking at other Holley 1940s online is the list number is the number 0120 under the 7860. It could be either but I'm not sure.
It runs great once its all warmed up, just while its warming up if you try to drive it and give it anymore than just a tap on the gas it will die, fall flat on its face. I tried playing with the choke being open or closed, closing it seems to make it worse.
Lik you said I'll rebuild it and maybe I'll find some other issue inside, or maybe setting back to an initial tune and going from there will help too, I have no idea how its set right now.

 

There actually is no port there, I was disappointed to not see it.