scrcpy do not work , scrcpy不能正常工作

[Sailiy]

hello，我添加了以下代码，
minicap可以正常工作

const minicap = deviceClient.minicap({});
    await minicap.start();
    minicap.on('data', (buf) => {
        let aa=cv.imdecode(buf)
        cv.imshow("aa",aa)
        cv.waitKey(1)
    })

scrcpy卡在 start()地方

const deviceClient = devices[0].getClient();
    const scrcpy = deviceClient.scrcpy({})
    await scrcpy.start()
    scrcpy.on('frame',(fram)=>{
        let aa=cv.imdecode(fram.data)
        cv.imshow("aa",aa)
        cv.waitKey(1)
    })

[UrielCh]

mmmm are you using my openCV code ?

what ever scrcpy send video frame, not JPEG buffer.

I also have a video support, but I it currently segfault on large video resolution. it mainly works in 480p....

tell me if you want to debug the video demux.

the demux code project is here: https://github.com/UrielCh/beamcoder

[Sailiy]

Yes, I used your @u4/ OpencV4Nodejs library and saved a lot of time on Windows, great; Thank you for your service.

My project needs to be integrated in Electron. Since I am not good at C/C++ programming, it is too difficult for me to use C/C++ in Electron, so I don't want to use C/C++ to complete decoding.

I have seen this project, which may also be enlightening to you, it uses both WebCodec and Tinyh264 decoding methods, which may be integrated into this project;

https://yume-chan.github.io/ya-webadb/

https://github.com/yume-chan/ya-webadb

[UrielCh]

Now I'm working on remote-droid
mostly precisely on its Kubernetes deployment.

Once remote-droid is completed I will improve scrcpy integration.

But the scrcpy frames must be demux before any openCV usage.
I tried to use it directly with OpenCV but it only supports hardware video input or video file input.
so there is no practical way for any real-time processing.

[Sailiy]

I've seen the scrcpy source code, and it's actually not that complicated to transfer a JPEG stream like minicap, which would eliminate a lot of decoding.
I looked at your Github Pages, you used to do Java, I thought you could try it

[Sailiy]

我大概写了一下代码，修改的文件时scrcpy项目server目录内的文件ScreenEncoder.java，由于android环境比较复杂，还没编译测试，我想可以解决。
有兴趣可以尝试一下

                    Image mImage=codec.getOutputImage(outputBufferId);
                    ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();
                    byte[] bytes = new byte[buffer.capacity()];
                    buffer.get(bytes);
                    Bitmap bitmapImage = BitmapFactory.decodeByteArray(bytes, 0, bytes.length, null);
                    ByteArrayOutputStream memStream = new ByteArrayOutputStream();
                    bitmapImage.compress(Bitmap.CompressFormat.JPEG,70,memStream);
                    byte[] buf=memStream.toByteArray();
                    ByteBuffer jpegBuf=ByteBuffer.wrap(buf);

[UrielCh]

yep but for that I need to maintain a scrcpy fork.
I have too many active projects. I can not hold that one.

I can integrate your fork if you want to deal with this subproject.

But patch it properly, it must be retro compatible with the original script.
since scrcpy V 1.20, it contain a real argument parser, add a flag for replacing stream video data with jpeg data, and we are good to go.

All STF features are available in scrcpy with better implementation.

[UrielCh]

You can check my fork: https://github.com/UrielCh/scrcpy
I do not like the memory management of this code, and I do not know if it can be built.

I do not have and java or android dev environment here.

And I did not write any line of java for years.

[tabearu]

Hey, do you have a running example on how to use scrcpy?
I need to stream the incoming video to multiple clients and save it on my server. For testing purposes I've tried

const adbClient = createClient();
const devices = await adbClient.listDevices();    
const deviceClient = devices[0].getClient();
const scrcpy = deviceClient.scrcpy();
const writeStream = fs.createWriteStream('./output.mp4');
await scrcpy.start()
setTimeout(async () => {
    scrcpy.stop();
}, 7000);    
scrcpy.on('frame', (frame) => {
    writeStream.write(frame.data);
})

but the output.mp4 always shows errors/ can't be opened.

Thanks!

[UrielCh]

check tasks/realTest.ts file, it contains test for most functions.

[UrielCh]

and use the last version the previous version had issue 9

[tabearu]

I've been testing some more and had a look at tasks/realTest.ts/extractFramesStream. I do get the 100 files, however I still can't view/open them, no matter which encoder for scrcpy I'm using. I'm currently using @u4/adbkit: 4.1.11 with two Android Devices running Android-Version: 10 and one other device running Android Version: 11. I also tried changing the scrcpy-jar version. Results are always the same.
Additionally I tried using the WebCodecs API to check if I can render the video on my client but with the same results. There's no errors thrown, but also no images/video.
Is there a good way to debug this?

[UrielCh]

I only succeeded in extracting images from this stream once using https://github.com/UrielCh/beamcoder
I think it's possible to render the stream directly in a browser, but I did not have success yet.

since I was able to identify keyframes and update frames, but I did not retire.

Do you know if I can build scrcpy on a windows host?

My Mac SSD is mostly full.

[tabearu]

Did you use beamcoder in a realtime-application or with a file? I'd like to start the stream from my backend, record the incoming scrcpy video there and then send it to the clients.
I also had a look at the repos mentioned by @Sailiy, which use the WebCodecsAPI to display the videostream in a canvas. They use the keyframe to define the canvas size (which works in my setup) and then use the remaining data for displaying (which does not work in my setup). I don't get any errors though and it's just an unfilled canvas.

I've downloaded the scrcpy zip with the download-link for windows, unzipped the folder and then defined the path with

const adbClient = createClient({ bin: `${config.SCRCPY_PATH}` });

which works for me.

[UrielCh]

If you want to integrate video from adbkit, you may be interested in remote-droid

Once fully deployed I get un GUI like:

Each server accepts multiple android devices, and all devices are visible at the same time as thumbnails

The front end is built in deno fresh. and I need some help to complete it.

Most of the configuration is done by Kubernetes.

[tabearu]

This sounds similar to what I'm trying to achieve. Just tested your docker image but unfortunately I can't see my devices. Maybe a problem because I'm using windows?

I only need scrcpy with one device and the video-stream should be recorded, before raw-data or a compiled video is sent to (multiple) clients. I need to continuously write the incoming data to a file because there's a high chance connection will be lost and I can't afford to lose all the data. Preferably this should be it's own service, so if there's a crash other services in my setup won't be affected.