Analytics (DWH) not working

OK, scratch that. I managed to mess it all up to the point where a fresh install was needed.

OS is CentOS 7 now, database is:

mysql Ver 15.1 Distrib 5.5.44-MariaDB, for Linux (x86_64) using readline 5.1

Still, the problem with reporting persists - analytics is not working, and this time around it doesn't display zero values - it simply shows the entire reporting screen in KMC greyed out. "Internal server error occurred" pops up the first time.

Sanity check fails during the DWH cycle with:

INSERT INTO csi_log VALUES(NULL,'DWH cycle','centos',1447519792,"Sent event [KalturaStatsEventType::WIDGET_LOADED] Sent event [KalturaStatsEventType::MEDIA_LOADED] Sent event [KalturaStatsEventType::PLAY] Sent event [KalturaStatsEventType::PLAY_REACHED_25] Sent event [KalturaStatsEventType::PLAY_REACHED_50] Sent event [KalturaStatsEventType::PLAY_REACHED_75] Sent event [KalturaStatsEventType::PLAY_REACHED_100] Executing [logrotate -f /etc/logrotate.d/kaltura_apache]Redirecting to /bin/systemctl reload httpd.service failed",255,11.770679758,'11.1.0');

I can do "service httpd restart" from the shell without any issues.

Meanwhile, the daily/hourly logs contain a single line:

sh: /opt/kaltura/bin/pentaho/kitchen.sh: No such file or directory

Here's grep for ERROR:

grep ERROR /opt/kaltura/dwh/logs/*
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,900 - Register server - Unexpected error
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,901 - Register server - org.pentaho.di.core.exception.KettleStepException:
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,902 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,902 - register_etl_server - Errors detected!
/opt/kaltura/dwh/logs/etl_hourly-20151114-09.log:ERROR 14-11 09:41:10,904 - Kitchen - Finished with errors

Ideas please?

Also, there are some error when running kaltura-run-dwh.sh

./kaltura-run-dwh.sh
reading config file /etc/logrotate.d/kaltura_apache

Handling 4 logs

rotating pattern: /opt/kaltura/log/kaltura_apache_errors.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_errors.log
log needs rotating
rotating log /opt/kaltura/log/kaltura_apache_errors.log, log->rotateCount is 5
dateext suffix '-20151114'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
destination /opt/kaltura/log/kaltura_apache_errors.log-20151114.gz already exists, skipping rotation
switching euid to 0 and egid to 0

rotating pattern: /opt/kaltura/log/kaltura_apache_access.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_access.log
log needs rotating
rotating log /opt/kaltura/log/kaltura_apache_access.log, log->rotateCount is 5
dateext suffix '-20151114'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
renaming /opt/kaltura/log/kaltura_apache_access.log to /opt/kaltura/log/kaltura_apache_access.log-20151114
compressing log with: /bin/gzip
switching uid to 0 and gid to 7373
running last action script
switching euid to 0 and egid to 0
Redirecting to /bin/systemctl reload httpd.service
switching euid to 0 and egid to 0

rotating pattern: /opt/kaltura/log/kaltura_apache_errors_ssl.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_errors_ssl.log
log /opt/kaltura/log/kaltura_apache_errors_ssl.log does not exist -- skipping
not running postrotate script, since no logs were rotated
switching euid to 0 and egid to 0

rotating pattern: /opt/kaltura/log/kaltura_apache_access_ssl.log forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 7373
considering log /opt/kaltura/log/kaltura_apache_access_ssl.log
log /opt/kaltura/log/kaltura_apache_access_ssl.log does not exist -- skipping
not running last action script, since no logs will be rotated
switching euid to 0 and egid to 0
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.get_data_for_operational does not exist
ERROR 1305 (42000) at line 1: PROCEDURE kalturadw.mark_operational_sync_as_done does not exist

Ok...having run this, now there's plenty in the DWH logs.

Daily log:

INFO 14-11 22:23:33,337 - daily - Starting entry [Check if update_dims_lock is seized]
ERROR 14-11 22:23:33,523 - Check if update_dims_lock is seized - An error occurred executing this job entry:
An error occurred executing SQL:
SELECT * FROM locks
WHERE lock_name = 'update_dims_lock' AND lock_state = 0
Table 'kalturadw_ds.locks' doesn't exist

update_dims:

INFO 14-11 22:23:29,619 - Kitchen - Start of run.
INFO 14-11 22:23:29,744 - Update Dimensions - Start of job execution
ERROR 14-11 22:23:29,927 - Update Dimensions - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log

Couldn't execute SQL: LOCK TABLES etl_log WRITE

Table 'kalturalog.etl_log' doesn't exist

INFO 14-11 22:23:29,928 - Kitchen - Finished!
ERROR 14-11 22:23:29,928 - Kitchen - Finished with errors
INFO 14-11 22:23:29,928 - Kitchen - Start=2015/11/14 22:23:29.620, Stop=2015/11/14 22:23:29.928
INFO 14-11 22:23:29,928 - Kitchen - Processing ended after 0 seconds.

Hourly:

INFO 14-11 22:23:29,619 - Kitchen - Start of run.
INFO 14-11 22:23:29,744 - Update Dimensions - Start of job execution
ERROR 14-11 22:23:29,927 - Update Dimensions - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log

Couldn't execute SQL: LOCK TABLES etl_log WRITE

Table 'kalturalog.etl_log' doesn't exist

INFO 14-11 22:23:29,928 - Kitchen - Finished!
ERROR 14-11 22:23:29,928 - Kitchen - Finished with errors
INFO 14-11 22:23:29,928 - Kitchen - Start=2015/11/14 22:23:29.620, Stop=2015/11/14 22:23:29.928
INFO 14-11 22:23:29,928 - Kitchen - Processing ended after 0 seconds.
[root@centos logs]# cat etl_hourly-20151114-22.log
INFO 14-11 22:23:25,849 - Kitchen - Start of run.
INFO 14-11 22:23:25,940 - hourly - Start of job execution
INFO 14-11 22:23:25,943 - hourly - Starting entry [Register etl_server]
INFO 14-11 22:23:25,944 - Register etl_server - Loading transformation from XML file [/opt/kaltura/dwh/etlsource//common/register_etl_server.ktr]
INFO 14-11 22:23:25,990 - register_etl_server - Dispatching started for transformation [register_etl_server]
INFO 14-11 22:23:26,076 - print_hostname_to_stream - Dispatching started for transformation [print_hostname_to_stream]
INFO 14-11 22:23:26,215 - Generate Row - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
INFO 14-11 22:23:26,219 - Mapping input specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,246 - Set hostname command - Optimization level not specified. Using default of 9.
INFO 14-11 22:23:26,328 - Set hostname command - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,331 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,332 - Select hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,333 - Mapping output specification - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO 14-11 22:23:26,334 - Get hostname - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
ERROR 14-11 22:23:26,350 - Register server - Unexpected error
ERROR 14-11 22:23:26,350 - Register server - org.pentaho.di.core.exception.KettleStepException:
Error in step, asking everyone to stop because of:

Error looking up row in database
Table 'kalturadw_ds.etl_servers' doesn't exist

    at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:307)
    at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
    at java.lang.Thread.run(Thread.java:745)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error looking up row in database
Table 'kalturadw_ds.etl_servers' doesn't exist

    at org.pentaho.di.core.database.Database.getLookup(Database.java:3120)
    at org.pentaho.di.core.database.Database.getLookup(Database.java:3093)
    at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:80)
    at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:290)
    ... 2 more

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'kalturadw_ds.etl_servers' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:389)
at com.mysql.jdbc.Util.getInstance(Util.java:372)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2535)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1911)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2034)
at org.pentaho.di.core.database.Database.getLookup(Database.java:3101)
... 5 more

INFO 14-11 22:23:26,351 - Register server - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
INFO 14-11 22:23:26,351 - register_etl_server - register_etl_server
INFO 14-11 22:23:26,351 - register_etl_server - register_etl_server
ERROR 14-11 22:23:26,351 - register_etl_server - Errors detected!
INFO 14-11 22:23:26,353 - hourly - Finished job entry Register etl_server
INFO 14-11 22:23:26,353 - hourly - Job execution finished
INFO 14-11 22:23:26,353 - Kitchen - Finished!
ERROR 14-11 22:23:26,353 - Kitchen - Finished with errors
INFO 14-11 22:23:26,353 - Kitchen - Start=2015/11/14 22:23:25.850, Stop=2015/11/14 22:23:26.353
INFO 14-11 22:23:26,353 - Kitchen - Processing ended after 0 seconds.

cat etl_perform_retention_policy-20151114.log

INFO 14-11 22:23:36,563 - Kitchen - Start of run.
INFO 14-11 22:23:36,650 - perform retention policy - Start of job execution
ERROR 14-11 22:23:36,833 - perform retention policy - A serious error occurred during job execution: org.pentaho.di.core.exception.KettleJobException:
Unable to begin processing by logging start in logtable etl_log

Couldn't execute SQL: LOCK TABLES etl_log WRITE

Table 'kalturalog.etl_log' doesn't exist

INFO 14-11 22:23:36,834 - Kitchen - Finished!
ERROR 14-11 22:23:36,834 - Kitchen - Finished with errors
INFO 14-11 22:23:36,834 - Kitchen - Start=2015/11/14 22:23:36.564, Stop=2015/11/14 22:23:36.834
INFO 14-11 22:23:36,834 - Kitchen - Processing ended after 0 seconds.