Creating a Group doesn’t automatically create a Forum. A Group can be attached to many kinds of activity, including a forum, but the activity has to be created separately and Groupings used for the attachment. You need to complete the following steps:
1 – Create the Groups you want to use, and add students to them. E.g. you could put all your students into teams, “Team1,” “Team2,” etc. You may use the “Auto-create groups” function to automatically divide all your students into groups of a specific size. This also allows you to automate step 2.
2 – Attach your Groups to a “Grouping.” In Course Administration->Users->Groups, click the “Groupings” tab and create a new grouping, e.g. “Teams.” Click the “Groups” icon in the rightmost column to add your groups “Team1,” “Team2,” etc. to this new Grouping. The idea here is that a Group is a list of individuals, but a Grouping is a collection of Groups that can be attached to a specific activity, whether or not they are going to share work cross-Group.
3 – Next, create the Forum (or other activity). (Return to your main course page. Be sure Editing is turned on. Go to the module or week of the course where you want the discussion to display, and use the “Add an activity…” menu to create a Forum.) In the “Common module settings” section, click “Show advanced.” If you want the groups to see each other’s postings, use Visible Groups. If you want them to see only their own group’s postings, use Separate Groups. Then pick the Grouping you created in Step 2 (“Teams”). This will present the activity to all students in the Grouping, but they will only be able to see and interact with other members of the Grouping as you have defined.
A student who is not in the assigned Grouping (not in any Group) will still be able to see the Forum link, but will not be able to see posts inside or add posts.
For more information, see the Moodle documentation on Grouping Users.