Remote Storage Only Not working

Hi @jlotito,

The “REMOTE STORAGE ONLY” option simply means that there will be no attempt to serve the content directly from the NFS in the event it was not found on the remote storage. The content will ALWAYS have to first be saved on /opt/kaltura/web/content because the source files need to be manipulated in various ways, mainly, different flavours should be created based on your transcoding profiles. Only after this is done, can the resulted files be exported to the remote storage.
Go to Admin Console->Batch Process Control and input an entry ID into the search box.
This will show the entire ingestion flow and all the jobs performed on a given entry.

There is an option in the Admin Console->partner->configure view called “Delete exported storage from Kaltura” but it does not seem to be working. I’ll try to allocate some time to debug this. Either way, you’d want to leave at least the source media file on the NFS because that’s needed in case you’ll want to reconvert, create thumbnails or perform other manipulations that require the source.

In regards to exporting the images [thumbnails] to the remote storage, there is such an option in the Admin Console->remote storage view, under the “Advanced” section. It is called “Should export thumbnail assets” but it does not seem to work. Again, I’ll try to allocate some time to debug this and understand why.

As to your question about creating thumbnails, one thumbnail will always be created when ingesting a new entry.
I would really advise against commenting pieces of code to disable the creation of a thumbnail, especially since you’ll have to manually patch it after every upgrade which is not a good idea.

That said, you can disable the creation of additional thumb assets by calling conversionprofile->update() on your default conversion profile and setting flavorParamsIds to ‘0,2,3,4,5,6,7’ instead of the default ‘0,2,3,4,5,6,7,19’.
19 is a flavour param called “Default Thumbnail”, the ID is a static one so it’s the same for all ENVs. When removing this ID from flavorParamsIds on the default conversion profile, the additional thumb asset you see under “Thumbnails” in KMC’s entry drill down view will not be auto created.

You can perform the update with a simple API script but you can also use the Kaltura CLI client for that.
To generate the CLI client, run this command on one of your Kaltura nodes:

# php /opt/kaltura/app/generator/generate.php cli

then:

# cd  /opt/kaltura/web/content/clientlibs/cli
# chmod +x setup.sh
# ./setup.sh </path/to/target/dir> <api_hostname> <partner_id> <partner_admin_secret>

Once that’s done, go to /path/to/target/dir and run:

$ kalcli -x conversionprofile list filter:objectType=KalturaConversionProfileFilter filter:systemNameEqual=Default ks=`genks -b $PARTNER_ID`

This will return something similar to:

KalturaConversionProfileListResponse
        objects array
                0       KalturaConversionProfile
                        id      9
                        partnerId       101
                        status  2
                        type    1
                        name    Default
                        systemName      Default
                        tags
                        description     The default set of flavors. If not specified otherwise all media uploaded will be converted based on the definition in this profile
                        createdAt       1515498358      (2018-01-09 06:45:58)
                        flavorParamsIds 0,2,3,4,5,6,7,19
                        isDefault       1
                        isPartnerDefault        1
                        cropDimensions  KalturaCropDimensions
                                left    -1
                                top     -1
                                width   -1
                                height  -1
                        clipStart       -1
                        clipDuration    -1
                        mediaParserType 0
                        calculateComplexity     1
                        collectionTags  mbr,ism
                        detectGOP       0
        totalCount      1

So, in the above example, the default conversion profile ID is 9.
Then, call:

$ kalcli -x conversionprofile update id=9 conversionProfile:objectType=KalturaConversionProfile conversionProfile:flavorParamsIds=0,2,3,4,5,6,7 ks=`genks -b 101`

Replacing 9 with your profile ID and 101 with your partner ID.

Regardless of your current question, the CLI client is a rather powerful tool when you need to perform simple API operations without writing code. It features tab completion for the services, actions and enums. You can learn more about how it works from the README under /opt/kaltura/web/content/clientlibs/cli/README.md

That brings us to your final question about a different FS structure for your media.

Before reading on, please understand that this operation should be carefully tested on a test machine before making any changes to your live production ENV.

In the Remote Storage profile configuration view, there is a selectbox called “Path Manager”.
Set that to “XSL Path” and go to the “Advanced” section.
There, in the “Path Manager Params (JSON)” textarea, you can input a JSON that contains a XSL describing an alternative structure.
For example, inputting this

[{"key":"path_format","value":"<xsl:value-of select=\"$partnerId\" \/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"php:function('date', 'Y', $currentTime)\"\/><xsl:text>-<\/xsl:text><xsl:value-of select=\"php:function('date', 'm', $currentTime)\"\/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"php:function('date', 'd', $currentTime)\"\/><xsl:text>\/<\/xsl:text><xsl:value-of select=\"$entryId\" \/><xsl:text>_<\/xsl:text><xsl:value-of select=\"$objectId\"\/><xsl:text>.<\/xsl:text><xsl:value-of select=\".\/content\/@extension\"\/>","relatedObjects":null}]

Would create the following structure under your remote volume’s root:

$REMOTE_STORAGE_ROOT/$PARTNER_ID/%Y-%m/%d/$ENTRY_ID_$FLAVOR_ID.$FILE_EXT

For example:

$REMOTE_STORAGE_ROOT/101/2018-01/10/0_mka2lp86_0_lfo55tdw.mp4

This will only take affect when uploading new entries, of course, the old will still need to be served from the original location. You can move them and then adjust the file_sync table but it’s a risky operation that should be carefully tested outside of production and I’d advise against it unless it is absolutely necessary.

Also, under “Advanced” you’ll see that you can choose which flavours to export to the remote storage [like I said the “Should export thumbnail assets” does not seem to be working at the moment but exporting only selected flavours does work, I tested]. Since you intend to serve all flavours from the remote storage, you should export them all but you can skip exporting the source flavour which would save some space on the remote volume. Depending on your needs, you may want to remove some flavours from the default transcoding profile or create several profiles you can choose from when ingesting an entry, naturally, the less flavours you have and the more compact they are, the less storage will be used.

In relation to conversion profiles and flavour params, you may find this thread instructive as well: