Liquid crystalline materials constituting anisotropic particles are known to exhibit self-assembled intermediate states between a translationally and orientationally ordered crystal phase and a disordered isotropic liquid phase. These intermediate states of these materials, known as mesophases, are characterized by a long-range orientational order with partial or no translational order. Some of these phases have been employed in various technological applications such as displays, sensors, lenses, phase modulators, etc. In addition to these applications, liquid crystal is a fertile area of research for fundamental studies on the nature of order-disorder transitions, symmetry-breaking phenomena, electro-optical effects, elastic instabilities, dynamics, and the nature of topological defects etc. Furthermore, the knowledge of liquid crystals can be utilized to understand other systems such as polymers, colloids, biological systems, and granular media wherein the orientational order can be observed. 

In this thesis, we study the self-assembled ordered structures in liquid crystalline systems, exploring the molecular organization, phase transitions, and instability-induced pattern formation. Various types of theoretical and computational tools, such as phenomenological theory, Monte Carlo simulations, and other numerical methods, have been employed to investigate these systems. 

In the talk, I shall discuss the effect of the excluded volume interaction on the spontaneous breaking of chiral symmetry in the tilted polar smectic phases of achiral bent-core molecular system. A phenomenological theory accounting for the smectic A -- de Vries smectic A -- smectic C phase sequence will be discussed. Furthermore, we have also constructed a phase-field model, namely time-dependent Ginzburg-Landau (TDGL) model C, to account for the banded spherulitic growth observed in the 8OCB liquid crystalline compound. I shall present the detailed studies on the TDGL model in the talk. A thin sheet floating on an isotropic substrate is known to exhibit wrinkling under unidirectional compression. I shall also present our theoretical investigations on the effects of the nematic substrate on the wavelength of the wrinkling instability.