Adding a digestive enzyme supplement does seem to work for some people, although when I tried to take fish oil, it never seemed to make any difference for me. I think it's worth a shot if you're determined to use the oil.
After reading Seth's post, I'm glad to know that the laxative effect of olive oil isn't just something idiosyncratic with my digestive system!
It makes me think of a Letterman show recently where chef Jamie Oliver was doing a cooking demonstration. Both Letterman and Tom Cruise started guzzling olive oil from the bottle, and Oliver said, "Someone alert the sewer department!"