It is very correct:) indeed if you set "Purge files on deletion" the case is different but you did not mention that you set it to true. By default, it is false.
Considering that it has been set to true, it should purge the files from the remote storage. If it does not, take a look at /opt/kaltura/log/batch/storagedelete*log and let us see what is in there.