The first library and reading room in Mountain View was opened to the public December 1, 1890. Money for the project was privately raised by local residents. On March 10, 1904, the Women's Club successfully petitioned the Town Board of Trustees for a Free Public Library and Reading Room; on April 14, 1905, the library opened above Rogers & Rogers general store with Mrs. Hessell as the Librarian. On June 7, 1905, the City of Mountain View took over the operation of the Library.

The 1906 earthquake caused damage to the Rogers & Rogers building. The library relocated, first to a room in a house on Hope Street, then a small house on Castro Street, and later an upstairs room in the Swall building on Castro Street.

Laurabelle "Dolly" Graham was appointed Town Librarian in 1907. She was born in Mountain View in 1856. Present day Graham Middle School was named after her father, Isaac Newton Graham. In her first month on the job, Graham reported a total of 446 books checked out and a count of 558 visitors to the Library.

April 9th, 1910, a new City Hall opened housing the Library on the first floor next to the city jail. In one account, the children's librarian said she could hear prisoners being interrogated in certain parts of the Library.

Graham resigned as Town Librarian January 1, 1915 after eight years of service, and Jessie Nichols succeeded her.

The City lost the opportunity to qualify for a Carnegie Library building in 1915 when officials declined to provide a site.

Arthur L. Palmer became Librarian in September 1919 and served until May 1944. Edna Roth was the appointed Librarian, and served until May 1957. Her successor was Edith Duke.

In 1952, the expansion of City Hall offices had made it necessary to find a new location for the Library. The Library moved to a converted store at 939 Dana Street. The collection totaled 17,000 volumes. In 1956, a $160,000 bond was passed for the first dedicated Library building, at 585 Franklin Street. This was an 8,400-square-foot building with a capacity of 65,000 volumes. The new building opened in the summer of 1958.

The Library began bookmobile service on April 28, 1964. A multi-level addition to the Library building was completed in 1968.

Ruth Lawson was appointed City Librarian in 1973, and served until 1981. She was succeeded by Ruth Stilwell, who retired in 1988. Susan Ozubko was then appointed Library Services Director

In June 1994, funding was approved for the construction of a new building at the 585 Franklin Street location. In May 1996, Library services moved to a temporary building on federally-owned land at the corner of Moffett Boulevard and Middlefield Road. The old building was demolished; the new building opened on October 25, 1997.

In 2001, Karen Burnett was selected to be the new Library Services Director. She served until December 2009. Rosanne Macek was hired in April 2010, and retired in July 2018. She was succeeded by Tracy Gray.

The Library began a remodeling project in 2017, and debuted its new facilities in June 2018, with an expanded Children’s Room; additional group study rooms; and another program room for Library events on the second floor.