Sequential Decision Making in Cooperative Multi-Agent Systems with Constraints