Former members of Congress worry mid-decade redistricting will amplify 'voices of the extremes'