Foundations of statistics research aims to establish fundamental principles guiding inference about populations under uncertainty. It is concerned with the process of learning from observations, notions of uncertainty and induction, and satisfying inferential objectives. The growing interest in predictive methods in high-stakes fields like medicine highlights the importance of such research in developing methods that provide interpretability and guarantees appropriate for high-stakes contexts. This thesis contributes advancements in Generalized Fiducial Inference, an inferential approach that obtains a distributional estimator of a parameter without the need for specifying a prior distribution. In particular, this thesis examines unifying results of the frequentist, Bayesian, and fiducial approaches to statistical inference. We establish asymptotic frequentist guarantees for a general class of generalized fiducial distributions under conditions similar to those demonstrated in the Bayesian setting. While asymptotic normality of generalized fiducial distributions has been studied before, this work significantly extends the usefulness of such a result by relaxing strict differentiability conditions. We demonstrate the applicability of our result with two examples that necessitate these more general assumptions: the triangular distributions and free-knot spline models. Additionally, we examine "foundational questions in data analysis" that arise in the context of analyzing data with complex underlying geometries. Such data complicates our choice of the fundamental unit of statistical analysis and suitable methodology. We demonstrate how Object Oriented Data Analysis facilitates a shape-based perspective that allows for more careful consideration of the data objects of interest and reveals more interpretable insights. We demonstrate this in two specific data examples: (1) quantifying 3-dimensional protein conformations and (2) functional data obtained from a biomechanical study of patients with knee osteoarthritis. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]