With his brother Tommy playing trombone, he formed Dorsey’s Novelty Six, one of the first jazz bands to broadcast. In 1924 he joined the California Ramblers (who were based in New York City). He did much free lance radio and recording work throughout the 1920s. The brothers also appeared as session musicians on many jazz recordings. He joined Ted Lewis's band in 1930, with whom he toured Europe.

After returning to the United States, he worked briefly with Rudy Vallee and several other bandleaders, in addition to the Dorsey Brothers Orchestra with Tommy. He appeared on at least seventy-five radio broadcasts (many with his brother), as a member of Nathaniel Shilkret's orchestra on programs such as the 1932 program, "The Music That Satisfies," also known as the Chesterfield Quarter Hour. Future bandleader Glenn Miller was a member of the Dorsey Brothers Orchestra in 1934 and 1935, composing "Annie's Cousin Fanny", "Tomorrow's Another Day", and "Dese Dem Dose", all recorded for Decca, for the band. Tommy broke off from the Dorsey Brothers Orchestra to form his own band in 1935 after a musical dispute with Jimmy. The Dorsey Brothers Orchestra became the Jimmy Dorsey Orchestra, and included musicians such as Bobby Byrne, Ray McKinley, and Skeets Herfurt along with vocalists Bob Eberly and Kay Weber.

In 1939 Jimmy hired Helen O'Connell as his female singer. She and Eberly possessed a "boy and girl next door" charm and their pairing produced several of the band's biggest hits. Many of the Eberly-O'Connell recordings were arranged in an unusual 3-section "a-b-c" format. The three-part format was reportedly developed at the insistence of a record producer who wanted to feature both singers and the full band in a single 3-minute 78 rpm recording. Eberly sang the first minute, usually as a slow romantic ballad, the next minute featured the full band backing Jimmy's saxophone, and the last minute was sung by O'Connell in a more up-tempo style, sometimes with lyrics in Spanish. Kitty Kallen sang with the Jimmy Dorsey orchestra following Helen O'Connell's departure in 1942.Jerry Lewis' first wife Patti Palmer (birth name Esther Calonico) was a singer with his orchestra for less than a year, starting about 1944.

Jimmy continued leading his own band until the early 1950s. In 1953 he joined Tommy's Orchestra, renamed "Tommy Dorsey and his Orch. featuring Jimmy Dorsey". On December 26, 1953, the brothers and their orchestra appeared on Jackie Gleason's CBS television program. The success of that television appearance led Gleason to produce a weekly variety program, Stage Show, hosted by the brothers on CBS from 1954 to 1956. Elvis Presley appeared on several of the telecasts. These were Presley's first appearances on national TV.

Jimmy took over leadership of the orchestra after Tommy's death. Jimmy survived his brother by only a few months and died of throat cancer, aged 53, in New York City. Broadcasts of Jimmy Dorsey and The Fabulous Dorsey Orchestra on NBC Bandstand survive from December 25, and December 31, 1956. At least two other extant broadcasts from the month of December 1956 are available as well. Recordings of the band from their winter 1957 tour have not surfaced. These recordings would provide the last aural evidence of Jimmy Dorsey's work. It is thought that Dorsey's last appearance was in Joplin, Missouri, on March 12, 1957

Shortly before his death, he was awarded a gold record for "So Rare" which was recorded on November 11, 1956. There is a controversy over who played the alto solo on the recording of "So Rare", Dick Stabile or Jimmy Dorsey. It reached the number-two spot on the Billboard charts, becoming the highest charting song by a big band during the first decade of the rock-and-roll era.

Jimmy Dorsey is considered one of the most important and influential alto saxophone players of the Big Band and Swing era, and also after that era. Jazz saxophonist Charlie Parker mentioned him as a personal favorite.

Jimmy Dorsey had eleven number one hits with his orchestra in the 1930s and the 1940s: "Is It True What They Say About Dixie?", "Change Partners", "The Breeze and I", "Amapola", "My Sister and I", "Maria Elena", "Green Eyes", "Blue Champagne", "Tangerine", "Besame Mucho", and "Pennies from Heaven" with Bing Crosby. In 1935, he had two more number ones as part of the Dorsey Brothers Orchestra: "Lullaby of Broadway" and "Chasing Shadows". His biggest hit was "Amapola", which was number one for ten weeks in 1941 on the Billboard pop singles chart. On August 17, 1936, Bing Crosby recorded "Pennies from Heaven" with the Jimmy Dorsey Orchestra, a recording that went number one for ten weeks and became one of the top records of 1936. And finally, there was a late hit in 1957, "So Rare", which went to the No. 2 position, and was on the record charts for 26 weeks.

Here is Jimmy Dorsey And Orchestra with Vocals by Bob Eberly From 1940. Enjoy