Yep - you have to look in the correct Library folder. There's at least 3 on any Mac system.
One in your user folder - the home folder.
One in the System folder,
and a Library (the one that you want) at the root of your hard drive
btw, the largest files that you are looking for might not be in the Audio folder, but will be in the Applications Support folder, where you should see a GarageBand folder.
Open the /Library folder, then select Show View Options from the View menu, and turn on "Calculate All Sizes", then click on the header to sort by file size.
You will quickly see where a lot of space is used, and the garageband files, such as the Instrument Library, and Apple Loops should quickly move to the top, so you can move them off. You may even find (if you have ever used Garageband) that there will be Logic folders with some of the files that Garageband might use.
If you don't use ANY audio apps, and don't expect to do that, then you can put most of those large Logic, or Garageband folders into the trash.
You can always just reinstall Garageband, in the event that you need those files in the future.
Don't forget to empty the trash, too