This is a bit tricky, but let me guide you through what is required.
You will need a custom login form which you add to a standard WordPress page using the wpv-login-form shortcode (https://toolset.com/documentation/user-guides/views-shortcodes/#wpv-login-form).
On the template which displays the post you want users to be able to edit, you insert the link to edit the post using your edit form as normal, meaning it will insert a toolset_edit_post_link shortcode.
Now, you need to use the toolset_access shortcode to control what Guest users can and can not see (as described here: https://toolset.com/documentation/user-guides/access-control-texts-inside-page-content/).
You will want non-Guest users (i.e. those who are logged in) to see the link to the edit form.
You will want Guest users to see a link to the page with the custom login form instead.
So your template where the edit link is inserted would look something like this:
[toolset_access role="Guest" operator="deny"]
[toolset-edit-post-link content_template_slug='edit-project-form-container' target='self']Edit %%POST_TITLE%%[/toolset-edit-post-link]
[/toolset_access]
[toolset_access role="Guest" operator="allow"]
<a href="[wpv-post-url item='138']">Login to edit</a>
[/toolset_access]
In that example, 138 is the ID of the page where the custom login form is inserted.
Now, this will achieve part of your task.
If you visit a post that uses that template, if you are logged in you will see the edit post link, if you are logged out you will see a link to login.
If you click the link to login you are taken to the custom login page—where you can log in—but then you are left on the same page.
The wpv-login-form has a redirect_url attribute to specify the page to redirect to after logging in. But on that page we don't know where to redirect to.
The solution is to pass the details of where to redirect to from the template that inserts the link to the page with the log-in form. When viewing a post on the front end which has the link to edit the post we do know what the URL we need to redirect to is: it is the same as the URL generated by the toolset_edit_post_link shortcode, which is the URL of the post to be edited, appended with a URL parameter content-template-id which specifies the ID of the content template which holds your edit form.
So, you need to update the link to the page with the custom login form to pass the post URL and the content template ID, like this:
<a href="[wpv-post-url item='138']?source=[wpv-post-url]&template=244">Login to edit</a>
The 'source' parameter will hold the URL of the post to be edited, and the 'template' parameter will hold the ID of the template which holds the edit form. (It is manually entered—as 244 in my example—and I got the ID by looking at the URL which the edit link generates.)
Now go back to the page where you insert the log-in form and make use of those two parameters to generate the redirect_url parameter of the wpv-login-form shortcode, like so:
[wpv-login-form label_username="Custom label for username/email" redirect_url="[wpv-search-term param='source']?content-template-id=[wpv-search-term param='template']"]
That is quite a lot to digest, but hopefully if you follow step-by-step you will be able to implement.
