As with many of these political arguments, I do not have a horse in the race only have some exposure to the massive Government structures. So let's look at a few Departments.
Department of Energy: Fundamentally it makes and houses nuclear weapons, the old AEC. Then all sorts of stuff was added. Forty years ago they added the electric car. Then along came Tesla and did in a year what DoE spent 40+ failing to do. So just dump all but the AEC stuff, move it to DoD. Close it down.
Department of Education: Well we have thousands of school boards, trillions of school taxes and a collection of politicos in DC disconnected from reality. Close it up.
HHS: Keep this one. It has FDA, NCI, etc.
Department of Agriculture: Now 150 years ago it made sense. Now not so much. Massive companies grow stuff now, so we would be better off managing it through FTC.
Department of Treasury: I guess we keep it.
Department of State: Now this is a real boondoggle, It made sense when communications was by sailing ship and before the President could pick up a phone or video conference. It should be cut in half at least!
Department of Interior: Well, parks and stuff. Outsource it.
DHS: Now here if the homeland police. They control not just borders but have empowered hundreds of thousands of agents to control our interstate travel. In many ways this may be the most destructive and abusive entity in the Federal Government. TSA is in my opinion incompetently and abusively run. But it seems we keep giving it more police powers, we need less!
DoD: Well this is a good one. In the old days the Army protected the country and the Navy protected us on the seas and in foreign ports. The Air Force was to do both. But 9/11 showed us that not a single aircraft or anti-aircraft could protect us from four straying aircraft. One wonders what Gatwick in the US would look like.
Treasury: Somebody had to raise the money for all this stuff.
EPA; Reason should prevail and people should be protected.
and the list goes on. It would be interesting to see what could really be done from the ground up again.