The Nazis established a civilian administration under SA Major General Hans Kramer. Between July and August 15, 1941, the Germans concentrated the remaining Jews, some 29,000 people, in a ghetto established in Slobodka. It was an area of small primitive houses and no running water. The ghetto had two parts, called the "small" and "large" ghetto, separated by Paneriu Street. Each ghetto was enclosed by barbed wire and closely guarded. Both were overcrowded, with each person allocated less than ten square feet of living space. The Germans continually reduced the ghetto's size, forcing Jews to relocate several times. The Germans destroyed the small ghetto on October 4, 1941, and killed almost all of its inhabitants at the Ninth Fort. Later that same month, on October 29, 1941, the Germans staged what became known as the "Great Action." In a single day, they shot 9,200 Jews at the Ninth Fort.

The ghetto in Kovno provided forced labor for the German military. Jews were employed primarily as forced laborers at various sites outside the ghetto, especially in the construction of a military airbase in Aleksotas. The Jewish council (Aeltestenrat; Council of Elders), headed by Dr. Elchanan Elkes, also created workshops inside the ghetto for those women, children, and elderly who could not participate in the labor brigades. Eventually, these workshops employed almost 6,500 people. The council hoped the Germans would not kill Jews who were producing for the army.

In the autumn of 1943, the SS assumed control of the ghetto and converted it into the Kauen concentration camp. The Jewish council's role was drastically curtailed. The Nazis dispersed more than 3,500 Jews to subcamps where strict discipline governed all aspects of daily life. On October 26, 1943, the SS deported more than 2,700 people from the main camp. The SS sent those deemed fit to work to labor camps in Estonia, and deported children and the elderly to Auschwitz. Few survived.

On July 8, 1944, the Germans evacuated the camp, deporting most of the remaining Jews to the Dachau concentration camp in Germany or to the Stutthof camp, near Danzig, on the Baltic coast. Three weeks before the Soviet army arrived in Kovno, the Germans razed the ghetto to the ground with grenades and dynamite. As many as 2,000 people burned to death or were shot while trying to escape.

Throughout the years of hardship and horror, the Jewish community in Kovno documented its story in secret archives, diaries, drawings, and photographs. Many of these artifacts lay buried in the ground when the ghetto was destroyed. Discovered after the war, these few written remnants of a once thriving community provide evidence of the Jewish community's defiance, oppression, resistance, and death. George Kadish (Hirsh Kadushin), for example, secretly photographed the trials of daily life within the ghetto with a hidden camera through the buttonhole of his overcoat.

The Kovno ghetto had several Jewish resistance groups. The resistance acquired arms, developed secret training areas in the ghetto, and established contact with Soviet partisans in the forests around Kovno. In 1943, the General Jewish Fighting Organization (Yidishe Algemeyne Kamfs Organizatsye) was established, uniting the major resistance groups in the ghetto. Under this organization's direction, some 300 ghetto fighters escaped from the Kovno ghetto to join partisan groups. About 70 died in action. The Jewish council in Kovno actively supported the ghetto underground. Moreover, a number of the ghetto's Jewish police participated in resistance activities. The Germans executed 34 members of the Jewish police for such activities.

The Soviet army liberated Kovno on August 1, 1944. Of Kovno's few Jewish survivors, 500 had survived in forests or in bunkers; the Germans evacuated an additional 2,500 to concentration camps in Germany.