The strong interaction of individual quantum emitters with resonant cavities is of fundamental interest for understanding light–matter interactions. Plasmonic cavities hold the promise of attaining the strong coupling regime even under ambient conditions and within subdiffraction volumes. Recent experiments revealed strong coupling between individual plasmonic structures and multiple organic molecules; however, strong coupling at the limit of a single quantum emitter has not been reported so far. In this study, we show that one can indeed observe strong coupling in the limit of a single quantum emitter. We use a unique technique to fabricate silver bowtie plasmonic cavities and couple them to semiconductor quantum dots (QDs). Scattering spectra registered from individual plasmonic cavities containing one to a few QDs show vacuum Rabi splitting, indicating that the strong coupling regime is approached in these systems. A coupling rate as high as 120 meV is registered even with a single QD, placing the bowtie-QD constructs close to the strong coupling regime. To obtain an unequivocal proof that the splitting in the spectra is due to strong (or close-to-strong) coupling of the bowtie longitudinal plasmon mode and the quantum emitter’s exciton, we resorted to polarization dependent experiments in which the polarization of the excitation source was rotated. When the excitation is polarized along the longitudinal direction of the bowtie (along its long axis), the spectrum shows two peaks due to Rabi splitting. As the polarization is rotated to the transverse direction, the double-peaked spectrum is replaced with the single-peaked spectrum of the transverse mode. These results, together with fits to the coupled oscillator model, electromagnetic calculations and photoluminescence measurements clearly indicate that the transparency dips in the scattering spectra are due to a genuine coupling of the plasmon and QD excitations.