Detail steps for this framework and issues

[AliceCodeZhang]

Could you please provide more detail description about how to use this framework ?
We got following issue when we running hybrid-vqa
Use tf.where in 2.0, which has the same broadcast rule as np.where
Epoch 1/50
2019-09-25 17:19:22.936808: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-25 17:19:23.222561: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
14/20 [====================>.........] - ETA: 4s - loss: 1.7395Traceback (most recent call last):
File "hybrid-vqa.py", line 102, in
steps_per_epoch=20, epochs=50)

ValueError: Error when checking input: expected time_distributed_3_input to have 5 dimensions, but got array with shape (2, 1)

Anything wrong ?

[malleshamdasari]

It says the expected and given input dimensions do not match. Can you recheck if you are passing the dimensions correct? It should be like (batch_size x no_of_frames x width x height x channels)

[WenyuanWu99]

It says the expected and given input dimensions do not match. Can you recheck if you are passing the dimensions correct? It should be like (batch_size x no_of_frames x width x height x channels)

How to do this ? I'm new about this, and I change the getData to change the datasource to KoNViD_1k_videos first.
getData.py:

# To add a new cell, type '#%%'
# To add a new markdown cell, type '#%% [markdown]'

# %%
import os
import numpy as np
import pandas as pd

# %%
set1 = os.listdir(
    '~/MachineLearning/KoNViD_1k_videos/KoNViD_1k_videos')


# %%
# mos1 = np.loadtxt('../deep-dataset/set1-mos.txt')
df = pd.read_csv('../deep-dataset/KoNViD_1k_attributes.csv')


# print(mos1)
print(df.head())


# %%
count = 1
tfile = open('../deep-dataset/mos.txt', 'w')


def getData(files, c):
    for f in files:
        os.system('cp ~/MachineLearning/KoNViD_1k_videos/KoNViD_1k_videos/'+f +
                  ' ../deep-dataset/traces/trace_'+str(c)+'.mp4')
        print(f, '../deep-dataset/trace_'+str(c)+'.mp4')
        # mosIndex = int(f.split('_')[1][:-4])
        # m = mos[mosIndex-1]
        mosValue = df[df['file_name'] == f]['MOS'].item()
        m = int(round(mosValue - 1))
        tfile.write('%s\n' % str(m))
        c += 1
    return c


count = getData(set1, count)

tfile.close()


# %%

Then I run getData and prepareData well. But I fail in deep-vqa and hybrid-vqa. I just change the input_shape from (200, height, width, 3) to (80, height, width, 3).

 alldata_x shape is  (2, 80, 68, 120, 3) 
 alldata_y shape is  (2, 5)
 5/20 [======>.......................] - ETA: 22s - loss: 1.7415Traceback (most recent call last):
  File "deep-vqa.py", line 97, in <module>
    steps_per_epoch=20, epochs=50)
ValueError: Error when checking input: expected time_distributed_1_input to have 5 dimensions, but got array with shape (2, 1)

Is anyone know sth about this?

[malleshamdasari]

It says the expected and given input dimensions do not match. Can you recheck if you are passing the dimensions correct? It should be like (batch_size x no_of_frames x width x height x channels)

How to do this ? I'm new about this, and I change the getData to change the datasource to KoNViD_1k_videos first.
getData.py:

# To add a new cell, type '#%%'
# To add a new markdown cell, type '#%% [markdown]'

# %%
import os
import numpy as np
import pandas as pd

# %%
set1 = os.listdir(
    '~/MachineLearning/KoNViD_1k_videos/KoNViD_1k_videos')


# %%
# mos1 = np.loadtxt('../deep-dataset/set1-mos.txt')
df = pd.read_csv('../deep-dataset/KoNViD_1k_attributes.csv')


# print(mos1)
print(df.head())


# %%
count = 1
tfile = open('../deep-dataset/mos.txt', 'w')


def getData(files, c):
    for f in files:
        os.system('cp ~/MachineLearning/KoNViD_1k_videos/KoNViD_1k_videos/'+f +
                  ' ../deep-dataset/traces/trace_'+str(c)+'.mp4')
        print(f, '../deep-dataset/trace_'+str(c)+'.mp4')
        # mosIndex = int(f.split('_')[1][:-4])
        # m = mos[mosIndex-1]
        mosValue = df[df['file_name'] == f]['MOS'].item()
        m = int(round(mosValue - 1))
        tfile.write('%s\n' % str(m))
        c += 1
    return c


count = getData(set1, count)

tfile.close()


# %%

Then I run getData and prepareData well. But I fail in deep-vqa and hybrid-vqa. I just change the input_shape from (200, height, width, 3) to (80, height, width, 3).

 alldata_x shape is  (2, 80, 68, 120, 3) 
 alldata_y shape is  (2, 5)
 5/20 [======>.......................] - ETA: 22s - loss: 1.7415Traceback (most recent call last):
  File "deep-vqa.py", line 97, in <module>
    steps_per_epoch=20, epochs=50)
ValueError: Error when checking input: expected time_distributed_1_input to have 5 dimensions, but got array with shape (2, 1)

Is anyone know sth about this?

Looks like your alldata_x shape is (2, 80, 68, 120, 3) is correct. But alldata_y shape is (2, 5) dimensions are not matching. Can you check that again?

[WenyuanWu99]

So what the alldata_y should be like?

[AliceCodeZhang]

alldata_y generate from mos file
def myGenerator(): while True: index_list = random.sample(range(1, 800), 2) alldata_x = [] alldata_y = [] for i in index_list: #print(i, index_list) f = videoFiles[i] s = f[:-4]+'_mos.npy' a = np.load(f) b = np.load(s) alldata_x.append(a) alldata_y.append(b) alldata_x = np.array(alldata_x) alldata_y = np.array(alldata_y) print(alldata_x.shape, alldata_y.shape) yield alldata_x, alldata_y

For each video frame , there is a corresponding mos.npy generated from prepare_data.py , the value is numpy array like this
[1. 0. 0. 0. 0.]

So, in my case alldata_y shape also print as (2,5)
my steps is

1. getData
2. PrepareData ->generate video frame and corresponding mos to numpy array
3. run hybrid-vqa to training the model
   Anything missing or wrong ?
   Thanks for your help.