The Atari 130 ST was a prototype for the press demonstration and was never released because of its small memory size: the complete operating system couldn't fit in it.
It was replaced by the Atari 260 ST and the Atari 520 ST. Despite its name, the Atari 260 ST was shipped with 512 KB of RAM, because the 192 KB operating system left less than 64 KB of free RAM.
The Operating System was TOS (Tramiel Operating System, later renamed just "The Operating System") which was in fact no more than the old CP/M 68K. Notice that to make easy the transition from the PC to ST, all the high level calls of the TOS (called GEMDOS) were compatible with the DOS calls of the PC (int 21h).
All the ST family systems used the GEM, which stands for Graphic Environment Manager, GUI (from Digital Research) which has a striking resemblance to the Macintosh GUI (windows, dialog boxes, desktop, desktop accessories, & bitmap fonts) even if it is a bit less friendly than the Apple one.
The 260ST Operating System (TOS & GEM) was supplied on diskette with earlier models, then into 6 x 32 KB ROM chips when the development of the O.S. was fully finished.
ST systems had several dedicated coprocessors for interrupt handling
(MFP 68901), sound (the old Yamaha YM 2149, same as General Instruments AY-3-8910 found in MSX computers, Amstrad or Oric), video (a custom chip called "Shifter"), and memory (a custom chip called "GLUE" and the MMU).