Activity-dependent GABAergic synapse plasticity is important for normal brain functions, but the underlying molecular mechanisms remain incompletely understood. Here, we show that Npas4 (neuronal PAS-domain protein 4) transcriptionally regulates the expression of IQSEC3, a GABAergic synapse-specific guanine nucleotide-exchange factor for ADP-ribosylation factor (ARF-GEF) that directly interacts with gephyrin. Neuronal activation by an enriched environment induces Npas4-mediated upregulation of IQSEC3 protein specifically in CA1 stratum oriens layer somatostatin (SST)-expressing GABAergic interneurons. SST+ interneuron-specific knockout (KO) of Npas4 compromises synaptic transmission in these GABAergic interneurons, increases neuronal activity in CA1 pyramidal neurons, and reduces anxiety behavior, all of which are normalized by the expression of wild-type IQSEC3, but not a dominant-negative ARF-GEF-inactive mutant, in SST+ interneurons of Npas4-KO mice. Our results suggest that IQSEC3 is a key GABAergic synapse component that is directed by Npas4 and ARF activity, specifically in SST+ interneurons, to orchestrate excitation-to-inhibition balance and control anxiety-like behavior.