CIS Seminar: “Intrinsic images, lighting and relighting without any labelling”

Intrinsic images are maps of surface properties. A classical problem is to recover an intrinsic image, typically a map of surface lightness,

from an image.   The topic has mostly dropped from view, likely for three reasons: training data is mostly synthetic; evaluation is somewhat

uncertain; and clear applications for the resulting albedo are missing.  The decline of this topic has a consequence – mostly, we don’t understand and can’t mitigate the effects of lighting.

I will show the results of simple experiments that suggest that very good modern depth and normal predictors are strongly sensitive to lighting — if

you relight a scene in a reasonable way, the reported depth will change. This is intolerable. To fix this problem, we need to be able to produce

many different lightings of the same scene.   I will describe a method to do so.  First, one learns a method to estimate albedo from images without any labelled training data (which turns out to perform well under traditional evaluations).  Then, one forces an image generator to produce many different images that have the same albedo — with care, these are relightings of the same scene.  Finally, a GAN inverter allows us to apply the process to real images.  I will show some interim results suggesting that learned relightings might genuinely improve estimates of depth, normal and albedo.