So i had exactly the same issue. Anything i'd glue to the backside of the foam panel, would fall off immediately or in a day. What i figured could be used is simple staples and adhesive tape.
I bought a box of industrial staples from homedepot, adhesive 3M mounting tape and made mini hangers our of them. You cut a strip of tape, then take couple of staples and put them through the tape so they look like a spikes. Then you stick the tape with these pins/spikes to the wall and simply hang foam panel on top.
Works perfectly! No damage to the wall or the panels. Could be removed or rearranged within seconds.
Made a quick how-to videos: https://youtu.be/svTCFVMheQ0