After installing Java for Mac OS X 10.5 Update 2 it’s no longer possible to set applet preferences in the Java Preferences application, including raising the maximum limit for heap memory usage.
This support article, Java for Mac OS X 10.5 Update 2: Setting Java Applet preferences, describes a workaround. See also this discussion thread for more details.