Multiple GPUs / HW device selection / FFStream decode debugging / nvidea eGPU / VDPAU

@Ig0r

I discussed this with the programmer and there is 1 possibility he saw looking at the code but we have no way to test this with any of our setups.  It would certainly be a shame to miss out on using your external GTX970. 

Do you have the capability of compiling Cinelerra from source code?  You do not have to be a rocket scientist to do this but just fairly computer knowledgable.  If so, there is a patch attached that was just generated that may provide the only possible solution and it will take a lot of time and some back and forth work for you to even test this.  How to compile from source code is documented in the manual which is on the cinlerra-gg.org website under documentation.

I will provide a little more information in a subsequent response but I have to look some things up first.  And yes, we would definitely be interested in your results if you can get this to work so we can pass it along to other users.

@Ig0r

Well the patch did not attach so I am listing it here but the formatting will probably mess it up.

--- ffmpeg.C.orig 2020-03-05 12:19:44.000360071 -0700
+++ ffmpeg.C 2020-04-12 12:34:25.825652597 -0600
@@ -1099,7 +1099,8 @@
if( hw_pix_fmt >= 0 ) {
hw_pixfmt = hw_pix_fmt;
avctx->get_format = get_hw_format;
- ret = av_hwdevice_ctx_create(&hw_device_ctx, type, 0, 0, 0);
+ const char *device = getenv("CIN_HW_DEVICE");
+ ret = av_hwdevice_ctx_create(&hw_device_ctx, type, device, 0, 0);
if( ret >= 0 ) {
avctx->hw_device_ctx = av_buffer_ref(hw_device_ctx);
ret = 1;

Could you disable HW_device in Settings, that is in Settings->Preferences, Performance tab change "Use HW Device" to "none". Quit out of Cinelerra, restart, and try to load your testvideo.MP4? And then check for terminal window messages. The reason I am asking is because of the following errors you received are odd as commented on next.

FFStream::decode: failed                            OK message and expected.
HW device init failed, using SW decode.     OK even though not what you want.
file:/home/ig0r/testvideo.MP4                      This is the file under question.
err: Invalid data found when processing input  WHY does the software report error?

Of concern is this last message. If you switched to SW (software) decode, then why would there be an error? Also, it might help if you could put your test video somewhere
for me to download because of the "Invalid data found when processing input" (or email
to me privately if you do not want it seen elsewhere to [email protected]). Perhaps it is just a problem caused by the previous attempt to process via hardware but it would be good to know for sure.

I have reviewed some of the information in your other forum as you suggested. The output of your vdpauinfo looks normal as far as I know. After talking to GG, the first output line may be more important than we currently realize:

ig0r@ig0r-ThinkPad-X230:~$ vdpauinfo
display: :0 screen: 0

As you noted in the other forum commentary "it seems to be an issue with ffmpeg" this
is consistent with what gg noted when he studied ffmpeg.c this morning. Currently only
the default "device" is used. The patch previously attached allows for setting a
specific device instead to pass to ffmpeg. It looks like CIN_HW_DEVICE environment
variable will have to be set after the patch is applied before starting up Cinelerra.
But I do not know what GG says it has to be set to in your case (but will have to ask
if you even get this far). When he looked at the documentation for ffmpeg, there was
information about changing the X11 display device so it really could be something like:

"It might be as simple as changing something like the "GPU number" inside some settings,"

So that instead of "display: :0 screen: 0" it would be "display: :1 screen: 0". Meanwhile
I will try doing some experiments on intel hardware that might have 2 graphics cards.

@PhyllisSmith, thank you so much for coming back to this so quickly, I really appreciate your efforts!

Concerning the self build: I'll give it a try ASAP, never did that before but I guess I'll manage. Not quite sure where exactly the provided patch (thx!! awesome!!) comes into play while building, but I didn't go through the whole build docu yet. I'll keep you posted on that one of course.

I'm happy to provide the testvideo - I'll share it via a google drive to your email, thanks for that! Also, I put various debugging screenshots there (since I'm not sure how to place them here). 

I did the test on setting the HW devide to "none" with the following output:

ig0r@ig0r-ThinkPad-X230:~$ cin
Cinelerra Infinity - built: Mar 31 2020 12:53:11
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

RenderFarmClient::main_loop: client started
FFStream::decode: Retry limit

So there is no complaint, but the playback is rather stuttery (compared to the VAAPI setting where at least the Intel "GPU" is being used) and the CPU is fully loaded too (so indeed SW decode mode).

I did some quick tests on changing the "video driver" in settings/preferences/*playback A from "X11" to "X11-XV" to "X11-OpenGL" and also playing around with the default display (from "" to "0" to "1")

The last line "FFStream::decode: Retry limit" btw I see always, no matter what HW device I use.

Correct me if I'm wrong, but I think mpv player also has FFMPEG as engine. If there would be an inherent issue with FFMPEG, it would also show up in using the mpv player I guess, which apparently doesn't generate any issues. 

In any case, thanks so much for helping me out on that one. And great if you could somehow test this on some hardware in parallel. But as I said I'm happy to test this with my setup as good as possible too.

All the best from Austria!

@PhyllisSmith, I just finished with my single-user build. I did the following, hopefully it was correct.

1. installed git and cloned the repository to a local drive (all via root)
2. The provided "patch" implementation was as simple as finding the ffmpeg.C inside the cinelerra subfolder of the cloned repository, making a copy just to be save (cp ffmpeg.C ffmpeg.C.orig) and deleting the lines starting with "-" and adding the lines starting with "+" in your patch code from above. The specific code section inside the ffmpeg.C now looks like this:

   .
   .
   if( hw_pix_fmt >= 0 ) {
   hw_pixfmt = hw_pix_fmt;
   avctx->get_format = get_hw_format;
   const char *device = getenv("CIN_HW_DEVICE");
   ret = av_hwdevice_ctx_create(&hw_device_ctx, type, device, 0, 0);
   if( ret >= 0 ) {
   avctx->hw_device_ctx = av_buffer_ref(hw_device_ctx);
   ret = 1;
   .
   .

3. Since this was my first build of cinelerra, I followed the commands in the documentation:

   ./blds/bld_prepare.sh ubuntu
   ./autogen.sh
   ./configure --with-single-user
   make 2>&1 | tee log
   make install

4. The make2>&1 | tee log took quite a while (approx 15 Min), as far as I can tell everything went smooth, I didn't find any errors in the last section of the output log. 
5. Start the build cinelerra via ./cin in the bin subdirectory -> and wow it really starts! 🙂 So I think the build was successful with the provided patch code.

However, the output log after loading the same testvide.MP4 gives similar results. First, I tried just via changing the performance settings (from blank to VDPAU to VAAPI) and again we get the "HW device init failed" output when I try to use VDPAU, and everything seems fine with VAAPI.

I also manually set the CIN_HW_DEV environment variable to vdpau, but with the same scenario:

root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# export CIN_HW_DEV=vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $CIN_HW_DEV
vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# ./cin
Cinelerra Infinity - built: Apr 13 2020 11:28:12
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: Retry limit
FFStream::decode: Retry limit

I noticed that apparently cinelerra tries to init the HW device more often now, maybe a sign that the patch is indeed implemented in this build? 

In any case, I think we did a step forward, even though the results are not quite there yet. At least I gained some know-how and can go forward with further patch implementations and single-user builds to debug this on my specific setup.

Let me know if you have some further ideas and how we should proceed best - looking forward to it!

All the best! Cheers!

EXCELLENT assist!  Thanks for sending the test video becaise with software only we also get a "retry limit" but it loads so gg can look at that.

OK, the CIN_HW_DEVICE environment variable must be set to your actual eGPU display hardware device.

If you are using the Display variable to get to your device:

for example we would use:  export DISPLAY=:0.0

for the primary display.  You should:  export CIN_HW_DEV=:1.0

for an alternate display on       display 1, screen 0.

@Ig0r

Clarification (sorry).  The environment variable should be CIN_HW_DEVICE, not CIN_HW_DEV.  So as not to conflict with the original.  The Settings->Preferences should still be set to vdpau.

@phyllissmith, thanks for the super fast follow up. I tried a couple of things now, all having the VDPAU in the HW device performance setting, but these two are the most representative I think:

root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $DISPLAY
:0.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# export CIN_HW_DEV=vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $CIN_HW_DEV
vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# export CIN_HW_DEVICE=:1.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $CIN_HW_DEVICE
:1.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# ./cin
Cinelerra Infinity - built: Apr 13 2020 11:28:12
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

Failed HW device create.
dev:vdpau
err: Unknown error occurred
virtual int FFStream::decode_activate():
open decoder failed
virtual int FFStream::decode_activate():
can't open input file: /home/ig0r/testvideo.MP4
Failed HW device create.
dev:vdpau
err: Unknown error occurred
virtual int FFStream::decode_activate():
open decoder failed
virtual int FFStream::decode_activate():
can't open input file: /home/ig0r/testvideo.MP4
Failed HW device create.
dev:vdpau
err: Unknown error occurred
virtual int FFStream::decode_activate():
open decoder failed
virtual int FFStream::decode_activate():
can't open input file: /home/ig0r/testvideo.MP4
FFStream::decode: Retry limit

It seems to me that we try to point to display 1, even apparently we use display 0. This fits to the output (as you pointed out already) of vdpauinfo:

ig0r@ig0r-ThinkPad-X230:~/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/cinelerra$ vdpauinfo
display: :0 screen: 0

Also the nvidia driver panel seems to state display 0 and only one screen, so I guess 0. I changed the numbers of CIN_HW_DEVICE=: to 1.0 / 0.1 / 1.1 but thats no good either.

So for me the straight forward things to do would be:

root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $DISPLAY
:0.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# export CIN_HW_DEV=vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $CIN_HW_DEV
vdpau
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# export CIN_HW_DEVICE=:0.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# echo $CIN_HW_DEVICE
:0.0
root@ig0r-ThinkPad-X230:/home/ig0r/cinelerra_singleuser_1/cinelerra5/cinelerra-5.1/bin# ./cin
Cinelerra Infinity - built: Apr 13 2020 11:28:12
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: avcodec_send_packet failed.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/home/ig0r/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: Retry limit

But I think it seems to be exactly what has been used as default, since the error messages are again the same. 

Looking forward to continue this topic whenever you have some new ideas, I'm happy to try.
All the best meanwhile!

I did not mean to imply that there was anything wrong with ffmpeg concerning your statement "If there would be an inherent issue with FFMPEG, it would also show up in using the mpv player".  It is just that Cinelerra interface to it may be a problem. (Although there are currently 9 patches that gg had to make to ffmpeg to get it to communicate to Cinelerra so there are problems).

Have not yet had time to do any more tests here but that is planned for today.  I am very happy to hear that following the Manual directions worked without any problems as things change fast and a bunch of users and me try to keep it up to date.

@phyllissmith, no problem if there is some delay. I'm anyway really stunned on the response time 🙂 

BTW let me know if there's a way to donate or buy you a beer, since I'm a big fan of this community based concept and I appreciate the time and efforts you guys but into this things.

@Ig0r

4 things:

#1 your test video if yuvj420p -- note the "j" (also frame rate 59 is unusual).You can see this by running:

$ ffprobe testvideo.MP4

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x4da5880] Using non-standard frame rate 59/1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'testvideo.MP4':
Metadata:
major_brand : mp41
minor_version : 538120216
compatible_brands: mp41
creation_time : 2020-02-29T13:29:01.000000Z
firmware : HD7.01.01.90.00
Duration: 00:00:11.03, start: 0.000000, bitrate: 60417 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p

#2 our testing on an intel device with 2 graphics  board shows below "Successfully created a VDPAU device" AND "Unsupported sw format: yuvj420p" so that is why vdpau does not work here and I am guessing for you also.

in libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x7f0de4026e80] Successfully created a VDPAU device (OpenGL/VAAPI backend for VDPAU) on X11 display :0
[AVHWFramesContext @ 0x7f0de7388900] Unsupported sw format: yuvj420p
[h264 @ 0x7f0de73a92c0] Failed setup for format vdpau: hwaccel initialisation returned error.
[h264 @ 0x7f0de73a92c0] Invalid return from get_format(): vdpau not in possible list.
[h264 @ 0x7f0de73a92c0] decode_slice_header error
[h264 @ 0x7f0de73a92c0] no frame!
[AVHWFramesContext @ 0x7f0de73bdfc0] Unsupported sw format: yuvj420p
[h264 @ 0x7f0de73c9300] Failed setup for format vdpau: hwaccel initialisation returned error.
[h264 @ 0x7f0de73c9300] Invalid return from get_format(): vdpau not in possible list.
[h264 @ 0x7f0de73c9300] decode_slice_header error
[h264 @ 0x7f0de73c9300] no frame!
[AVHWFramesContext @ 0x7f0de73b3800] Unsupported sw format: yuvj420p
[h264 @ 0x7f0de73b8d40] Failed setup for format vdpau: hwaccel initialisation returned error.
[h264 @ 0x7f0de73b8d40] Invalid return from get_format(): vdpau not in possible list.
[h264 @ 0x7f0de73b8d40] decode_slice_header error
[h264 @ 0x7f0de73b8d40] no frame!
[AVHWFramesContext @ 0x7f0de73b3800] Unsupported sw format: yuvj420p
[h264 @ 0x7f0de7426dc0] Failed setup for format vdpau: hwaccel initialisation returned error.
[h264 @ 0x7f0de7426dc0] Invalid return from get_format(): vdpau not in possible list.
[h264 @ 0x7f0de7426dc0] decode_slice_header error
[h264 @ 0x7f0de7426dc0] no frame!
FFStream::decode: avcodec_send_packet failed.
file:/tmp/testvideo.MP4
err: Invalid data found when processing input
FFStream::decode: failed
HW device init failed, using SW decode.
file:/tmp/testvideo.MP4

#3 I forgot to relay this.  In your Cinelerra directory under the subdirectory ffmpeg, there is a file called decode.opts which is used during the load video operation.  Change the line "loglevel=fatal" to "loglevel=verbose".  Restart Cinelerra from the command line and then load testvideo.MP4 and you should see messages similar to the above.  Do you see "Successfully created a VDPAU device" with X display 0 and "unscupported format yuvj420p"?  If so, then whichever graphics card is in use (Display 0, I guess from you vdpauinfo output) does not support yuvj420p.

#4 We are now building Cin on a Fedora computer that has a GTX970 and will see if it does support yuvj420p -- it is not clear from the vdpauinfo output (yours and ours on an intel computer) if that is supported.

@Ig0r

Sam (@Sam) maintains the website in Europe and it is not set up to take donations but perhaps he could use a "virtual beer" -- you would have to ask him.  GG and I are not interested in donations of any kind because we just do this for fun and do not want to feel obligated to work on any specific thing.

@Ig0r

We do not work for profit, but on a voluntary basis for the common good, and so do I. Nevertheless we want to offer our community the possibility to spend a beer or a pizza for the server costs and infrastructure. We are working on making this possible in the future as well.

One more thing.  Our intel computer vainfo reports the following (see below) supported devices and you can see on the last line that JPEGBaseline (which we are assuming includes yuvj420p) is supported.  Your vainfo does not include this.  So vaapi works on our computer.  I can not understand why vaapi works on your computer but am guessing that it is using the "other" graphics board?

$ vainfo

Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264MultiviewHigh : VAEntrypointVLD
VAProfileH264MultiviewHigh : VAEntrypointEncSlice
VAProfileH264StereoHigh : VAEntrypointVLD
VAProfileH264StereoHigh : VAEntrypointEncSlice
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
VAProfileJPEGBaseline : VAEntrypointVLD

When we booted a Fedora computer with only the one GTX970 board, it also reports that yuvj420p is not supported.  Vaapi does not work to load your test video on that single board computer.

I have not studied your last note yet but will do so.

MY CONCLUSION - the only problem is that yuvj420p is not supported by your Graphics card.  And again am "guessing" that it has been using your GTX970 board all along (based on your previous note that I glanced through, where Display 0 really is your eGPU.