You can talk to Dr Banner just fine. It's just occasionally the other guy comes out and smashes up half a city in a murderous indestructible rage.
Likewise you can talk to Skye just fine. Until, y'know, she causes earthquakes in your underground base through methods you don't understand in the slightest. They've every reason to be cautious.