you're getting caught up on the easy part of the story lol.
Columbia is the brain child of booker (who becomes Comstock after he finds god) after he backs luticia (spelling). From the tears he sees from her experiments he sees his future (which is really just possibilities of his future and why he calls himself the prophet) and sees his heir. Because of the experiments though he's gone sterile, so in order to have "his child" become his heir he goes to another of his realities, the one where he isn't baptized, and steals his alternate selfs baby. Booker gives up Anna because he lost a large amount gambling (his addiction because he still has the guilt from all the things he's done in war). Fast forward, the twins (which is really the same person but from different realities) set in motion a plan to stop Comstock/Elizabeth from baptizing the world in fire, that is basically the entire premise of the game. In the end, it took I believe 123 times of the events in the game to happen before the little changes and choices in each time to add up to enough to let booker not only save Anna (he would previously always die to songbird), but to also realize the truth himself and stop the cycle of booker being baptized and becoming Comstock. Elizabeth only says she's not his daughter because she doesn't know of any of this until the end of the game, all they know is lady Comstock isn't the mom, and eventually that Comstock isn't the "father".
Now, rapture itself doesn't have anything specific to do with Columbia, it's just one of the things they saw through the tear. But because of it, the Fitz brothers were able to steal and modify the big daddy process to create songbird to protect Elizabeth. Also, and this is inferred because it's never mentioned, they also stole the tech for vigors based off plasmids from rapture. They also stole many other things in general from the tears; music is mentioned, which people loved (obviously because it's loved in its original reality) and it's insinuated that that's where finks fortune comes from.