Assistant/add upload file option

[rosannamilner]

This removes the previous fixed list of video and image tools and replaces with StringFileUploadField component to either paste URL or upload an image or video to submit to the Assistant.

• Behaviour of close buttons etc copied to be the same as the other tools
• TODO decide where to put archive button for archiving URLs (it should only appear if urlMode === true)
• TODO rethink tooltips and intro description
• TODO store image/video for a URL to send to the DBKF similar image/video service

On opening the Assistant:

InVID Translations: AFP-Medialab/InVID-Translations#144

[Sallaa]

Hi @rosannamilner Do you want to reuse the component we use for tools with the URL and file upload? This could also be used.

Also, we have made tabs in our UI that was using the "two big buttons" to pick between URL and local file in the tools such as CheckGIF and Keyframes. Could you have a look too? I am not sure these UIs are totally relevant with what the file upload you are trying to make, but reusing it could save you some time.

[rosannamilner]

Hi @rosannamilner Do you want to reuse the component we use for tools with the URL and file upload? This could also be used.

Also, we have made tabs in our UI that was using the "two big buttons" to pick between URL and local file in the tools such as CheckGIF and Keyframes. Could you have a look too? I am not sure these UIs are totally relevant with what the file upload you are trying to make, but reusing it could save you some time.

Thanks Valentin - I'll take a look and it makes sense to reuse the URL and file upload component, it does feel a bit clunky having these two big buttons to start with rather than directly letting user choose.

So the upload section is for a user to upload their own image or video and see the "Extracted media files" card with just the single media present. The user would see the Recommended tools list appear next to the image/video. The existing code has the media stored as blob files which can then be displayed.

To send the media file off to the DBKF similar images service, it looks like I need to actually store the file somewhere. I tried converting the file to a data URL but the example image I tried makes the URL too long and the DBKF service returns 414 error code. Given the DBKF code/api requests are not on the backend, can these uploaded files be stored on the frontend? How is it done for the other tools? The DBKF service expects a URL to the image/video.

[Sallaa]

If I understand correctly @rosannamilner DBKF does not allow to send a file (upload file) and always requires an URL. We do not really have this problem with other APIs because they can receive files. Does using the form component break things for you or was it already a problem before?

If they only accept URLs, I see mainly 2 options: 1) skip the DBKF call for local files because they only accept urls, 2) have a way in the sheffield back-end to make a temporary url which means uploading the file to your back-end.

[rosannamilner]

If I understand correctly @rosannamilner DBKF does not allow to send a file (upload file) and always requires an URL. We do not really have this problem with other APIs because they can receive files. Does using the form component break things for you or was it already a problem before?

If they only accept URLs, I see mainly 2 options: 1) skip the DBKF call for local files because they only accept urls, 2) have a way in the sheffield back-end to make a temporary url which means uploading the file to your back-end.

Thanks Valentin - I will check whether their API will receive files, otherwise work something out in the backend.

For this PR I will skip the DBKF call (the StringFileUploadField component seems to work, I will do a bit more testing) and create a separate issue as it may take longer.

[rosannamilner]

Not all the recommended plugin tools accept the blob format - looking into what we can do in the backend with @ianroberts

[ianroberts]

2. have a way in the sheffield back-end to make a temporary url which means uploading the file to your back-end.

This is probably the way we'll go - rather than the plugin directly calling the DBKF service, we'll add a kind of proxy layer in our backend so that the user can POST images to us, we host them at a temporary URL which the backend passes to DBKF, the backend receives the response from DBKF, sends it back to the client and then deletes the temporary uploaded file.

[rosannamilner]

This is probably the way we'll go - rather than the plugin directly calling the DBKF service, we'll add a kind of proxy layer in our backend so that the user can POST images to us, we host them at a temporary URL which the backend passes to DBKF, the backend receives the response from DBKF, sends it back to the client and then deletes the temporary uploaded file.

The other issue I'm having is with the recommended tools for the uploaded media. Clicking on a tool sends the blob URL again, which doesn't work when submitted to that tool as a URL or as a local file (tried for image Magnifier and Metadata tools so far). Is there something I'm missing here, or an alternative option? Could the temporary URL provided by our backend be used across the tools (but hidden to user if necessary)?

[ianroberts]

How many of those tools have a "push" API option where you upload the data to them, rather than them pulling it down from a URL? Dealing with local image files is a fundamentally different operation from dealing with URLs to remotely-hosted files.

[ianroberts]

I'm thinking a fairly simple way to handle the handing off of an already-uploaded File from the assistant to the other tools could be to introduce a new context and a couple of custom hooks - I'm thinking something like this in Hooks/useUrlOrFile.jsx:

import React, {createContext, useContext, useReducer, useState} from "react";
import {useSearchParams} from "react-router-dom";

const defaultSelectionContext = {
    /** @type {string | undefined} */
    url: undefined,
    /** @type {File | undefined} */
    file: undefined,
    /** @type {(item?: (string | File | undefined)) => void} */
    setAssistantSelection: () => {},
}

const AssistantSelectionContext = createContext(defaultSelectionContext);

// if reducer needs to return an empty object, always use the same one
const emptyState = {};

export const AssistantSelectionProvider = ({children}) => {

    const [{url, file}, setAssistantSelection] = useReducer((state, action) => {
        if (typeof action === "undefined" || action === null) {
            return emptyState;
        }
        if (typeof action === "string") {
            // avoid re-render if setAssistantSelection is called with the same value twice
            return state.url === action ? state : {url: action};
        }
        // else it must be a File - again avoid re-render if setAssistantSelection is called
        // with the same value twice
        return state.file === action ? state : {file: action};
    }, {});

    return (
        <AssistantSelectionContext.Provider value={{url, file, setAssistantSelection}}>
            {children}
        </AssistantSelectionContext.Provider>
    )
}

export const useUrlOrFile = () => {
    const {url, file} = useContext(AssistantSelectionContext);
    const [params] = useSearchParams();
    const fromAssistant = params.has("fromAssistant");

    // if we were redirected from the assistant then use the context
    // values as the initial defaults for our state
    const [myUrl, setUrl] = useState(fromAssistant ? url : undefined);
    const [myFile, setFile] = useState(fromAssistant ? file : undefined);

    return [myUrl, setUrl, myFile, setFile];
};

export const useSetInputFromAssistant = () => {
    const { setAssistantSelection } = useContext(AssistantSelectionContext);
    return setAssistantSelection;
}

You'd need to insert the <AssistantSelectionProvider> somewhere high enough up the tree to cover both the assistant itself and the other tools to which we want to be able to hand off, either with all the other context providers in AppWrapper or a bit deeper in MainContent. Then any tool that uses a StringFileUploadButton to ask the user for its input you would replace the pair of states with a call to useUrlOrFile:

  // instead of
  const [input, setInput] = useState(resultUrl);
  const [imageFile, setImageFile] = useState(undefined);

  // use
  const [input, setInput, imageFile, setImageFile] = useUrlOrFile();

Now in the assistant you can get the updater function with const setAssistantSelection = useSetInputFromAssistant(), and when you want to delegate to one of the other tools you just say

setAssistantSelection(fileInput ?? formInput);
navigate("/path/to/tool?fromAssistant");

[ianroberts]

I suppose it might make sense to lift the actual fileInput and formInput states from Assistant up to the context too, so that if the user presses the back button to go back to the assistant it has the URL or file they had previously selected, rather than resetting the state, so they can then forward the same input to another tool.

[ianroberts]

In a way this approach is kind of re-inventing the redux wheel - we could just put the File in the redux state and useSelector in the useUrlOrFile hook, but you're not really supposed to put non-serializable things into a redux store (though we already do this anyway with the setUrl functions in selectCorrectActions - really we ought to just put the tool names in the state and then pull the right object out of tools.jsx by name when we need to access its assistantProps etc.)