It's called a 'buffer cache' because it buffers the i/o data on its way to/from the disk. When an App writes data it first will be deposited into the Apps file buffer memory region and will subsequently be requested via library routines to have the kernel (the OS) copy it from the App's buffer to disk. The kernel will oblige and will copy it first to its buffer -- the file system buffer cache. If the kernel requires more room in its buffer cache it will obtain it from the free memory. When this happens the free memory value, in say the Terminal's top command, will immediately show a reduction of free memory. At some later point the kernel will copy this data (referred to has dirty buffers) to the appropriate disk location. I believe the frequency of this being done is 30 secs -- called sync-ing to disk.
As the usage of X increases with time without rebooting the kernel file system buffer cache will fill with the most needed or most frequently used data. This should help explain why some people claim that the system appears to perform better the longer they've been running X. The needed data for doing things (or maybe most of it) is now all resident in memory (the kernel's buffer cache) and doesn't need to be read from disk. This is much much faster.
As mentioned above, the kernel will expand its buffer cache on demand by using the free or unused memory in the machine. This explains that with time (could be a short period of time or a long period of time -- it depends on system usage/workload) the system appears to be using all of the available RAM per the Terminal's top command.
One other point to make is that if the kernel's buffer cache has grown to be quite large and is consuming a large percentage of the installed RAM there's no harm being done. If a new App is launched the kernel will release as much its buffer cache as needed. First it will release parts of the buffer cache that aren't 'dirty' until it figures it can honor the new App's memory demand. If by releasing all the non-dirty buffer segments it still requires more memory for the App then it will start writing the dirty segments of the buffer cache to disk and releasing their memory which in turn can be given to the new App. This stops when all the memory required by the new App is satisfied. In this manner the kernel buffer cache shrinks down in size. There's probably a minimum size to which it will shrink down to. In this case the kernel will start looking for other memory that's inactive. This could be a dormant App's memory. In this case the kernel will start to page out the dormant App's memory hoping to satisfy the new App's memory requirements. This kernel activity is called paging or swapping.
