This is one of the few documented parts of system tuning under Solaris. Have a look at "Solaris 2.3 Administering Security, Performance, and Accounting", page 201. Just put a
set msgsys:msginfo_msgmnb=<value>in your systems file. Be prepared that there is a unfixed bug in Solaris which masks the value you give with 0xFFFF thus truncating it to less than 64k. You may fix this by applying the following binary patch:
[Text with leading "->" is user input, with "::" is the expected response.] -> adb -w /kernel/sys/msgsys -> msgget+0x74,5?i :: msgget+0x74: ld [%l2], %l2 :: sll %l2, 0x10, %l2 :: ld [%fp - 0x8], %l1 :: srl %l2, 0x10, %l2 :: st %l2, [%l1 + 0x3c] -> msgget+0x78?W0x1000000 :: msgget+0x78: 0xa52ca010 = 0x1000000 -> msgget+0x80?W0x1000000 :: msgget+0x80: 0xa534a010 = 0x1000000 -> msgget+0x74,5?i :: msgget+0x74: ld [%l2], %l2 :: nop :: ld [%fp - 0x8], %l1 :: nop :: st %l2, [%l1 + 0x3c] -> $q