Funny, how you sometimes dont realize that you have a major bug right in front of your eyes.

The bounding box of the gameworld cut off the buildings at the second floor and made it all very dark.

Removed the bounding box with a new one made in blender instead of one side unity blocks.

Now the buildings are fully visible and also the sun is visible.