In order to use wheel scrolling event with GLUT on Mac, you should patch the GLUT source code and compile it. The Mac-version GLUT source code is available here on Apple’s developer website. After unzipping, modify or patch the source code appropriately referring to this page. In my MacBook Pro running Lion, I had to use the old Xcode 3 to compile the code. I modified the project settings so that it uses GCC 4.0 as the C/C++ compiler instead of the default GCC 4.2, and rolls back the base SDK to the old Mac OS X 10.5. Then I release built the code and replaced the resulting framework with the existing /System/Library/Frameworks/GLUT.framework after making a backup of course ;).
Now it should send the mouse *up* events having button values of 3 or 4, respectively for wheel up and down.