The TS832 FPV transmitter has an output of 600mw and will get pretty hot during normal use, especially if the quad isn't flying. With these higher-power transmitters people sometimes add a small finned aluminium heatsink to help dissipate the heat better. I've done this a few times with my quads and it does help, particularly when the FPV transmitter is mounted within an enclosed shell.
The connections are fairly straightforward, there are lots of wiring diagrams available online and in normal usage it's quite difficult (but still possible) to get it wrong. The maximum input voltage is 12.6v, so if you're running off a P2 battery voltage of 11.2 volts (3S), that should be quite OK and there is also a 12v or VBAT output from the VTX to power a camera. Your wiring colour-coding seems to be a bit of a mix-n-match, so re-check your connections once more. Where does the red wire, the one with the pink heatshrink run to?... I believe that's the +ive for the 12v out to camera. Where it goes to is not quite clear from the photo... You also have both ground wires connected to something?
But if the LED display isn't lighting up then you must have some sort of problem. Are you absolutely sure that you didn't accidentally power it up with no antenna attached, even momentarily? If you did, then you may have cooked it and will have to buy another.
It's also possible that you have a duff one straight from the factory .The easiest way to test it for correct operation is probably to connect it all up with simple wiring loom and a battery/power supply. You can connect a camera if you want - but it's not essential to make any camera connections because all you're interested in knowing is whether there is an blank-screen output signal.
I'm sure there must be other advice that can be offered but that's all I can think of so far. The photos don't really help that much because we can't clearly see what runs to where - so a circuit diagram, even hand-drawn, and some better pics would be quite useful.