The New York hedge fund firm recovered $70 million in losses from a fraudulent scheme connected to the fast-growing business of funding legal proceedings. Now its managers are telling others to tread with caution in this niche market.